Skip to content
Canonical

ID-64 — Historic re-ingest doc review (S341, 11/06/2026) — {64.7}/{64.8} inputs

Historic re-ingest doc review — distilled inputs for {64.7} retention + {64.8} cutover runbook

Section titled “Historic re-ingest doc review — distilled inputs for {64.7} retention + {64.8} cutover runbook”

Status: review complete 11/06/2026. Six historic documents reviewed against the CURRENT canonical-pipeline architecture (cocoindex on-prem stack → fresh Supabase preview branch → prod cutover). Every doc predates the canonical pipeline — the first re-ingestion arc (S175–S188, April 2026) ran on the now-retiring scripts/kb_pipeline/ TS/Python path. This review extracts what SURVIVES as {64.7}/{64.8} input and marks the rest SUPERSEDED so it is not re-litigated. Full per-doc analysis preserved in the session record; this doc is the durable distillation.

1. The six documents

Section titled “1. The six documents”
#DocWhereDateOne-liner
1staging-refresh.mddocs-site runbooks/05/05/2026Staging branch reset + reference-data re-seed runbook
2re-ingestion-status-review-2026-04-21.mdarchive operations/21/04/2026Post-cutover snapshot of the FIRST re-ingestion arc (S182)
3re-ingestion-skill-notes.mdarchive operations/S186–S188Raw decision logic + live execution gotchas from the first arc
4guide-regeneration-prompts.mdarchive operations/16/04/20269 guides (117 sections) + 13 MCP-authored items, recreation prompts
5blank-db-restore-matrix.mddeleted — recovered git show 26d0d884b^ (archive)16/04/2026FK-ordered 25-step restore sequence; 5-category disposition per table/entry-point
6database-rebuild-runbook.mddeleted — recovered git show 26d0d884b^ (archive)pre-S165End-to-end DB rebuild from migrations (reset → push → types → seed → verify)

Docs 5+6 were removed in the Phase-3 docs relocation; recover via the archive repo’s git history if needed verbatim.

2. What SURVIVES → {64.7} retention-plan inputs

Section titled “2. What SURVIVES → {64.7} retention-plan inputs”
  • The five-category disposition framework (doc 5): Replayable-from-source / Re-seedable / Manual-recreate / Export-then-reimport / Lost-acceptable. Use as the classification vocabulary for every table in the {64.7} plan (it composes cleanly with the RESEARCH §6 RETAIN/RE-SEED/MIGRATE/RE-INGEST/DROP set).
  • Reference-data restore list (doc 1 §4.4 — canonical 11-table pg_dump/restore set): taxonomy_domains, taxonomy_subtopics, taxonomy_sync_state, layer_vocabulary, guides, guide_sections, entity_aliases, company_profiles, template_requirements, feed_flags, tag_morphology_drift_flags — excludes user_roles (auth FK) and feed_prompts (created_by FK), seeded separately. Verify current names before use (template_requirements → form_template_requirements post-T2; new tables since — e.g. form_types, procurement_vehicles — extend the list per RESEARCH §6.1.)
  • Guide + MCP-content restoration targets (doc 4): 9 guides / 117 sections (4 sector + 3 product + 1 company + 1 auto-generated research guide) and 10 restoration-worthy MCP-authored content items (7 consolidation items + Telehouse + SI brief + case study; 3 demo items lost-acceptable). Under the fresh-preview-branch model, guides are RETAIN = export-then-restore; the regeneration prompts are the FALLBACK only. A/B verification queries included in doc 4 §D.
  • Keep-set artefact exclusion: the first arc proved quality baselines and keep-set queries must exclude title LIKE '[E2E%' / '[SUPERSEDE%' test artefacts. Fold into the {64.7} keep-set query.
  • Workspace re-creation (this session’s OQ-64-7 resolution): export the three intelligence-workspace configs (purpose, company-profile binding, guide, relevance_threshold), re-create on the fresh target, delete GNEWS-DEDUP.
  • content_history: clean slate (OQ-64-9, this session) — the doc-3 v1-history backfill machinery is NOT carried; the only pre-step is the client-action UUID sweep.

3. What SURVIVES → {64.8} cutover-runbook inputs

Section titled “3. What SURVIVES → {64.8} cutover-runbook inputs”

Skeleton (doc 5 restore order + doc 6 steps, adapted to preview-branch model): pre-flight → snapshot/exports → provision fresh preview branch (migrations replay — Task 82 proved this) → types regen + parity CI → auth/user seed + pipeline service-account verification → reference-data restore → the ID-45 cocoindex run → post-run verification gates → cutover → smoke → rollback plan.

Operational gotchas that carry forward verbatim:

  • Session-mode pooler (port 5432) for any migration touching auth.* — export SUPABASE_DB_URL_DIRECT for that invocation only (doc 6 §4).
  • Pipeline service-account health probe (doc 6 §7): SQL probe on a0000000-0000-4000-8000-000000000001 expects (token_null=false, identities=1); bun run seed:e2e-users --check exit 2 = S156 drift → idempotent corrective push.
  • pg_dump --data-only emits no SET search_path — prepend it or fully-qualify; use the Postgres-17 binaries (/opt/homebrew/opt/postgresql@17/bin/) against 17.x.
  • Protected-branch reset is a silent no-op via MCP — supabase db push --linked only.
  • Long-running ingest invocations need nohup … & disown (+ PYTHONUNBUFFERED=1) — plain & died by SIGHUP mid-corpus in S187.
  • Schema-parity diff post-provision: diff information_schema.columns old↔new (S187 precedent: 44 tables / 605 columns / 0 diffs = green).
  • Cutover exercises rarely-fired paths — post-cutover triage = dev-tools console across 5–10 surfaces, not just a landing-page check; screenshot parity audit (4-agent browser baseline old→new→diff) caught 2 real regressions at the first cutover.
  • Rollback patterns: structural failure → restore-from-snapshot; content-quality failure → mass-supersede (preserves history). Decide which classes trigger which in the runbook.
  • Wipe-rebuild fallback only (if a preview branch is ever NOT used): DROP SCHEMA public CASCADE does not drop extensions (move/drop vector first), storage policies survive and need a DO-block drop, and migrations pre-seed config tables that need TRUNCATE CASCADE before any data restore.

Verification-gate concept (docs 2+3): the first arc’s quality gate ran 8 dimensions (6 must-pass: chunk coverage, classification coverage ≥85%, guide-domain-filter resolution, embedding coverage, …). The old quality-gate.ts profile is the SEED for the equivalent gate over cocoindex output — this is an ID-45 spec-chain input, not {64.8} scope (the runbook references the gate; ID-45 defines it for the new write surface, including the new tables — reference_items, q_a_extractions, relationship triples per ID-101).

4. SUPERSEDED — do not carry (re-litigation killers)

Section titled “4. SUPERSEDED — do not carry (re-litigation killers)”
Historic itemSuperseded by
All EP1–EP11 replay commands (import_bid_library.py, ingest_markdown.py, ingest.py, batch-reclassify, backfill-chunks)The cocoindex run owns ingest end-to-end (ID-45); chunking at ingest ({56.8} RecursiveSplitter); old path retires under ID-46
Vercel env-repoint / Supabase project-transfer cutover (Path A+)Fresh preview branch → prod cutover model (OQ-64-8, Task 82); mechanism detail = {64.8} design point
compare-quality.ts id-pairing + title fallbackNew-pipeline verification = ID-45 spec input (above)
v1 content_history backfill + trigger machineryOQ-64-9 clean slate
feed_articles→content_items re-link yield expectationsO4 reference layer: reference_item_id backlink is the forward model; only the LEGACY retained-rows re-link remains, scoped in {64.7}
entity_relationships legacy counts/backfill framingID-101 owns relationship capture at parity (C1 gate)
Uppercase-domain classifier fix, post_insert.py no-op, Google-News resolvedUrl fix in kb_pipelinekb_pipeline is retiring (ID-46); equivalent concerns live in cocoindex extraction (bl-93 port landed; ID-101 parity eval) — only the Google-News opaque-URL rows matter as a {64.7} keep-set data note
Stage-1/Stage-2 Union-vs-client-only gate (Gate 3)ID-45 re-spec; corpus composition decided in the ID-45 chain

5. Open items surfaced (for Liam / routed)

Section titled “5. Open items surfaced (for Liam / routed)”
  1. Guide + MCP-content restore source ({64.7}): are SQL dumps of guides/guide_sections (+ the feed config) preserved in data/exports? Which of the three standalone source docs (Telehouse PDF, SI-brief docx, case-study transcript) survive in Storage/local? Determines export-restore vs prompt-recreate.
  2. coverage_targets intent (carried open since S182): intentionally empty (feature inert until authored in-app) or missing seed? Affects {64.7} disposition.
  3. Keep-set artefact exclusion ({64.7}, ratify-by-default): exclude [E2E-*] / [SUPERSEDE-*] titles + the 3 demo items from any retained-content export.
  4. Cutover mechanism detail ({64.8} authoring): preview-branch-verified corpus → prod — by data restore (pg_dump of verified branch into prod) or by branch promotion? The old project-transfer answer is dead; decide in {64.8}.
  5. Quality-gate equivalent for the cocoindex run → route as ID-45 spec-chain input (seeded with the old 8-dimension must-pass profile).
  6. Legacy Google-News rows: 8 retained legacy content_items carry opaque news.google.com/rss/... source_urls — disposition in the {64.7} keep-set (re-resolve, keep-as-is, or drop).
  7. products-services taxonomy domain still active? Verify under ID-63 CV load — one-line check, not a new task.