ID-64 {64.7} — Retention export / re-link runbook (pre-re-ingest)

Model: source-driven (bl-106 v2 RATIFIED S350). The keep-set is which SOURCE corpus feeds the cocoindex walk — RE-WALK / MIGRATE-CARRY / DROP per provenance — NOT a content_items row-preservation list. Corrected derivation: bl-106-keepset-v2-source-driven.md.

{64.7} Retention export / re-link runbook (Spine S3)

Scope. This is a runbook, not a migration — NO DDL is authored here. It states which client SOURCE corpus feeds the clean full-corpus re-ingest (ID-45), HOW the small MIGRATE-CARRY tail (rows whose content_items row is the only surviving artefact) is exported and materialised as source files staged into the walk, and HOW feed_articles is re-linked after the cocoindex pipeline mints fresh uuid5 ids. DDL column drops ({64.12}/{64.13}) and the cutover data-move ({64.8}) are owned elsewhere; this runbook produces the source-corpus input that the ID-45 walk consumes.

Source-driven model (ratified — do NOT re-decide). Per ID-45 INV-6, ALL nine write targets — including content_items and q_a_extractions — are truncated to zero before the walk; no old row survives. The walk (INV-8/9) runs over source files + URLs and regenerates content_items fresh (new uuid5 ids, classification + embedding + extraction at ingest). The keep-set therefore answers “which source corpus feeds the walk?”, not “which rows to preserve”.

Provenance is NULL on re-ingest BY DESIGN. The cocoindex content branch (CONTENT_ITEMS_SCHEMA / ci_target.declare_row, scripts/cocoindex_pipeline/flow.py lines ~1139-1168) writes id, content, title, content_type, embedding, op_id, source_document_id, primary_domain, primary_subtopic — and NOT ingestion_source (verified: the only two ingestion_source writes in flow.py belong to REFERENCE_ITEMS_SCHEMA, the ID-75 pullmd/URL reference_items branch). So 100% of re-ingested content_items rows carry ingestion_source = NULL; the legacy provenance values (qa_import, python_markdown, url_import, rss_feed) are retired by the re-ingest. Re-ingest provenance is instead carried by op_id + audit_log + source_documents (op_id landed T8 migration 20260521203414, stamped per-row by the flow). (Optional/minor follow-up, NOT {64.7} scope: have the content branch stamp a sentinel ingestion_source such as 'cocoindex_reingest' so the read-path enum can validate — route to ID-45 / pipeline owners; the runbook does not depend on it.)

Provenance grounding. Authored against the {64.7} ledger journal (ratified decisions encoded, NOT re-decided), the corrected source-driven census (real PROD data, 631 rows, queried 13/06/2026 — bl-106-keepset-v2-source-driven.md), ID-45 PRODUCT INV-6/8/9, ID-64 RESEARCH §6 + phase-a-findings + phase-b-execution-plan, and flow.py (CONTENT_ITEMS_SCHEMA, localfs.walk_dir). De-identified throughout — aggregate counts only, generic “the first client / handover DB”; no client codename appears anywhere.

Column-name currency (verified in supabase/migrations/): the legacy provenance column is content_items.ingestion_source — renamed from ingest_source by migration 20260612093515 (ID-107.1). It is read here only to bucket old rows by provenance for the RE-WALK / MIGRATE-CARRY / DROP decision; it is NOT carried forward (NULL on re-ingest, above). The handover-DB schema column-drops ({64.12} = 5 AI-telemetry token cols; {64.13} = notes, parent_id, source_bid, source_document) affect the fresh target schema, not the read used here.

0. Source corpus + the uniform local-fs walk

The client SOURCE corpus lives at Documents/development/local-fs (Coolify-mounted into the pipeline container) and is ingested uniformly via connectors.localfs.walk_dir(live=True, recursive=True) (flow.py docstring, step 1 — verified). There is NO separate python ingest path: the python_markdown rows’ source is just markdown files in local-fs, re-walked like any other file; binary documents are extracted via Docling on the same walk. Everything the walk regenerates comes from files (or re-fetched URLs) staged into local-fs — never from old content_items rows.

The keep-set is therefore expressed as three dispositions over the existing PROD corpus, keyed by the legacy ingestion_source bucket (read-only, for routing only):

RE-WALK — the source is still walkable; stage it into local-fs, let the walk regenerate it fresh. Old rows truncated (INV-6), NOT carried.
MIGRATE-CARRY — the content_items row is the only surviving artefact; export its source content and materialise it as a source file staged into the local-fs walk corpus, so the re-walk regenerates it cleanly.
DROP — never staged into the source corpus (test artefacts / opaque / smoke).

1. Source-corpus staging (RE-WALK)

The RE-WALK classes are staged into local-fs and regenerated by the walk; their old content_items rows are truncated (INV-6) and not carried.

legacy bucket	rows	source location → walk treatment
`qa_import`	440	client Q&A source corpus (client-provided files), staged into `local-fs` → `extract_qa_form` → `q_a_extractions` regenerated fresh. 70% of the corpus — its source availability is the #1 re-ingest dependency (D2).
`url_import`	55	re-fetch via pullmd on the walk where the URL is alive. ID-110 owns `url_import → reference_items` (referenced, not owned here). Dead URLs fall to MIGRATE-CARRY (§2).
`python_markdown`	23	markdown files re-walked from `local-fs` if the `.md` sources exist; else MIGRATE-CARRY (§2). Source availability UNCERTAIN — phase-a Item 1 found only 3 standalone source docs surviving (`kh-client-content-archive` + git `b6e35683d^`).

D2 dependency. Confirm the client’s Q&A source corpus (440 rows, 70%) is physically available and staged into local-fs for the walk. This is the single biggest re-ingest dependency; the walk cannot regenerate what is not staged.

2. MIGRATE-CARRY export (the small carry tail)

The genuine carry tail = content_items rows where the row is the only surviving artefact (no re-walkable source file/URL):

in-KH-authored manual (≈23 non-artefact; 10 of these archived — per-row authored-in-KH vs uploaded-file split needs confirmation, census D3);
any python_markdown whose .md source is gone (§1);
legacy rss_feed content with a dead source URL — iff Liam later opts to keep it (default disposition for rss_feed is MIGRATE-CARRY-or-DROP; the feed_articles re-link in §3 applies either way).

Procedure. Export the SOURCE content of each carried row, then materialise it as a source .md file staged into the local-fs walk corpus so the re-walk regenerates it cleanly (new uuid5 id, fresh classification/embedding):

1
-- {64.7} MIGRATE-CARRY export. Run vs PROD handover DB (read-only).
2
-- Carries SOURCE content only; the carried rows are materialised as .md into local-fs.
3
-- :CARRY_SOURCES = the carry-tail buckets (default: in-KH 'manual'; + 'python_markdown'
4
--   / 'rss_feed' only where the source is dead — confirm per §2).
5
SELECT id,                       -- export-time correlation only; NOT re-imported as PK
6
       title, content, content_type,
7
       source_url, source_file,  -- source_document (text) dropped {64.13}; values were
8
                                 --   migrated to source_file ?? source_url by 20260428131822
9
       created_by, content_owner_id, created_at
10
FROM   public.content_items
11
WHERE  ingestion_source IN (:CARRY_SOURCES)
12
   AND title NOT LIKE '[E2E%'                          -- DROP-58 exclusions carry into the
13
   AND title NOT LIKE '[SUPERSEDE%'                      --   carry export (verbatim quality-gate.ts:510)
14
   AND ingestion_source IS NOT NULL                      --   (the 2 NULL smoke stubs are DROP)
15
   AND NOT (source_url LIKE '%news.google.com/rss%');    --   (the 8 gnews rows are DROP)

EXPORT (source columns only): title, content (the body materialised into the .md), content_type, source_url / source_file, created_by, content_owner_id, created_at (authorship/ownership lineage). id is carried for the re-link key map (§3) and audit ONLY — it is not re-imported (the walk mints a fresh uuid5).

DISCARD (regenerated by the walk — never carried): classification_* (domain/subtopic/confidence/reasoning/model/cache), embedding + embedding_model, quality_score (+ *_updated_at / previous_*), summary / summary_data / ai_keywords / suggested_title, freshness*, citation_count, content_text_hash (generated column). Telemetry token columns are dropped by {64.12} regardless.

Store the carry export + materialised .md files under the staged source-corpus location (git-ignored — never committed; the client corpus stays out of the repo, phase-a Item 1). De-identify any filename (no client codename). Materialise-as-source, NOT in-place UPDATE — the carried rows re-enter via the walk as fresh content; old derived columns are never carried.

3. `feed_articles` re-link (the C3 renumbering problem)

feed_articles is the per-workspace RSS ledger (OQ-64-2 CLOSED) — RETAIN structure, do not truncate. Its content_item_id (nullable uuid) links a passed article to its promoted content_items row. The re-ingest mints new content_items ids, so every non-null content_item_id becomes a dangling pointer (C3 renumbering).

Re-link procedure (after the re-ingest mints new ids):

Retain feed_articles rows whose linked content is in the keep-set (the 32 non-gnews rss_feed survivors); rows linked to dropped content get content_item_id = NULL (or are dispositioned with their dropped gnews link).
Build an old→new id map from the export (§2 retained id) by matching survivors to freshly-minted content_items on a stable natural key — primary key: source_url (feed_articles.external_url ↔ re-ingested content_items.source_url); fallback: exact title match where source_url is absent/ambiguous.
Re-point feed_articles.content_item_id to the new id via the map. Rows that fail both source_url and title match are reported (not silently nulled) for manual disposition.

feed_sources / feed_prompts are RETAIN (re-ingest re-reads source configs + relevance-prompt lineage; feed_articles.prompt_version_id stays valid).

4. `content_history` — clean slate at handover (OQ-64-9)

Ratified: content_history is a CLEAN SLATE at handover (v1-history backfill machinery is NOT carried). One mandatory pre-truncate sweep first:

Client-action sweep. Identify rows representing real client actions — keyed by client user UUIDs in created_by (i.e. created_by NOT IN (<system/service UUIDs>), e.g. excluding the trigger default a0000000-0000-4000-8000-000000000001). Expected minimal (most rows are trigger-written v1 auto_v1_on_insert / initial_ingest audit rows from ensure_v1_history_at_commit, 20260612093515).
Disposition explicitly BEFORE truncate — record the swept client-action rows (count + ids) in the {64.7} export bundle so nothing real is lost silently; only after the sweep is logged does content_history go to clean slate.
Then clean-slate content_history (live count ≈ 1,225 → cleared). The fresh pipeline re-establishes v1 history on re-ingest insert via the existing trigger.

5. Workspace re-create / delete (OQ-64-7)

Workspace CONCEPT/PURPOSE carries (not the rows verbatim): on the fresh target, re-create the 3 intelligence workspaces — MAT Auditing, NHS Digital Cyber Alerts, Education Sector Monitor — with the same purpose / config (company-profile binding, guide, relevance_threshold) sourced from current-live config (phase-a Item 1: export-from-current-live, not historic-dump). DELETE the GNEWS-DEDUP workspace (not re-created). Path-A content is workspace-AGNOSTIC (content_items has no workspace_id); only Path-B form-writes + feed_articles resolve a workspace_id, so the 3 re-created workspace ids feed the feed_articles re-link (§3) and Path-B writes.

6. RETAIN-table preservation list (do NOT truncate)

Carried through the re-ingest unchanged — truncating any of these breaks the run:

entity_aliases and entity_pair_resolutions — admin-curated Stage-5 input + determinism cache; truncating breaks Stage-5 idempotency (OQ-64-4, ratified).
user_roles (drives get_user_role() → all RLS) · user_profiles (FK target for every *_by/owner column) — mandatory RETAIN.
company_profiles (single real client/org profile) · feed_sources · feed_prompts · feed_articles structure (re-linked, §3).
Reference-data restore set (canonical pg_dump/restore — verify current names, several renamed post-T2): taxonomy_domains, taxonomy_subtopics, taxonomy_sync_state, layer_vocabulary, guides, guide_sections, form_template_requirements (was template_requirements pre-T2), feed_flags, tag_morphology_drift_flags. Taxonomy / controlled-vocabulary disposition is owned by ID-63, referenced here.

Clean-slate write targets (truncated to zero pre-walk per INV-6, then repopulated by the ID-45 run — NOT this runbook): content_items, source_documents, q_a_extractions, q_a_pairs, q_a_pair_history, entity_mentions, entity_relationships, content_chunks, pipeline_runs, form_templates, form_template_fields, reference_items.

7. Validation step

Validate the source-staging plan before the walk runs:

Provenance census (routing oracle) — SELECT ingestion_source, COUNT(*) FROM content_items GROUP BY 1 ORDER BY 2 DESC vs the corrected census: qa_import 440 / manual 71 (23 real + 48 artefacts) / url_import 55 / rss_feed 40 (32 + 8 gnews) / python_markdown 23 / NULL 2 = 631 (census drift → escalate, do not proceed).
DROP-58 excluded from staging — confirm 48 [E2E%/[SUPERSEDE% + 8 gnews (source_url LIKE '%news.google.com/rss%') + 2 NULL-source smoke stubs are NEVER staged into local-fs (631 − 58 = 573 rows are source-relevant). Prod bracket-family census is exhaustive (other_bracket_family = 0).
RE-WALK source availability — confirm the qa_import Q&A corpus (D2, the #1 dependency) and any RE-WALK python_markdown / url_import sources are staged into local-fs / re-fetchable; anything missing falls to MIGRATE-CARRY (§2) or is escalated.
MIGRATE-CARRY materialised — each carried row (§2) has a corresponding .md staged into the walk corpus; the carry export carries the DROP-58 exclusions.
Post-walk regeneration check — after the walk, content_items is fully regenerated (new uuid5 ids; ingestion_source = NULL across the board); the count reflects the staged source corpus, NOT 573 (rows can split/merge under the new extraction).
Re-link coverage — every retained feed_articles row resolves to a new content_items id via §3 (any unmatched row reported, not nulled).

Pass condition: the DROP-58 set is excluded from staging, the RE-WALK sources (esp. the qa_import corpus) are confirmed staged, the MIGRATE-CARRY tail is materialised as source .md, and the post-walk content_items is fully regenerated with NULL provenance. There is no “573-row keep” oracle under the source-driven model — 573 is only the count of source-relevant old rows (631 − 58 DROP), not a row-preservation target.

8. Outstanding / pending

D2 — qa_import source corpus (the #1 re-ingest dependency). Confirm the client’s Q&A source corpus (440 rows, 70%) is physically available and staged into local-fs for the walk. The walk cannot regenerate un-staged source.
D3 — MIGRATE-CARRY per-row split. Confirm which manual rows are authored-in-KH (carry as source .md) vs uploaded-file (RE-WALK from local-fs), and whether any python_markdown / rss_feed rows have a dead source needing carry.
Optional/minor follow-up (NOT {64.7} scope). The content branch stamps no ingestion_source (100% NULL on re-ingest). Stamping a sentinel (e.g. 'cocoindex_reingest') so the read-path enum can validate is a latent write-path gap — route to ID-45 / pipeline owners; the runbook does not depend on it.
FYI cross-ref (NOT {64.7} scope). The corpus-wide q_a_extractions → q_a_pairs promotion (ID-45 INV-23) is currently unowned — tracked at the ID-45 / ID-59 seam ({59.14} covers form-responses only, not the corpus-wide promotion).
De-identification reminder. Export filenames + any logged/materialised sample must carry no client codename; aggregate counts only, consistent with the ID-45 specs’ “first client / handover DB” framing.