Re-ingestion Quality Protocol

Re-ingestion Quality Protocol

Last updated: 14/04/2026 (Plan D D5/D6/D7 — Session 168 WP1c) Owner: Knowledge Hub operators Status: Ready to run. All three scripts referenced below have landed on main (WP1b, S168). The Python truncation divergence (Divergence 1 below) was closed in the same session.

---

1. Purpose

This is the quality gate for re-ingestion of the Knowledge Hub content corpus. It defines a six-step protocol — plus a bonus embedding smoke test — for measuring whether a format or pipeline change has improved, degraded, or held constant the quality of the stored content, classifications, entities, and embeddings.

The protocol is invoked whenever the content-extraction pipeline is changed in a way that could alter stored content, embeddings, or derived classifications — most commonly a re-ingestion after the plain-text-to-markdown migration or after a taxonomy change. It is the operator-facing companion to the spec in .planning/.archive/.specs/content-format-standardisation-spec.md (SS6, SS7; archived S168 — all phases shipped). For the end-to-end rebuild sequencing, see database-rebuild-runbook.md.

---

2. Prerequisites

• bun install complete (for the TypeScript scripts).
• python3 -m pip install -r requirements.txt complete (for the Python pipeline).
• .env.local populated with NEXT_PUBLIC_SUPABASE_URL, SUPABASE_SECRET_KEY (or SUPABASE_ANON_KEY), and OPENAI_API_KEY.
• At least 2 GB of free disk in the repository root — snapshot files include the full 1024-dim embedding vector per item and routinely reach hundreds of megabytes.
• data/ directory is gitignored at the repository root. Confirm with git check-ignore data/ before writing snapshots there.
• For scripts that talk to Supabase via @supabase/supabase-js, invoke Bash with dangerouslyDisableSandbox: true. This is the standard CLAUDE.md workaround for the Bun 204 issue with the sandbox proxy.

---

3. Protocol steps

Step 1 — Snapshot current state

Run the snapshot script to capture every row of content_items alongside the derived fields used in the comparison (heading count, chunk count, entity canonical names, embedding).

bun run scripts/snapshot-content-state.ts

Default output: data/snapshots/pre-reingest-YYYY-MM-DD.jsonl. Override with --output. Use --limit N for a sanity-check run and --no-embeddings for a lightweight snapshot that omits the 1024-dim vector field (useful when you only need classification stability but not embedding stability).

Flags, per bun run scripts/snapshot-content-state.ts --help:

Flag	Purpose
--output PATH	Destination file (default data/snapshots/pre-reingest-YYYY-MM-DD.jsonl)
--no-embeddings	Skip the 1024-dim vector field
--limit N	Limit total rows (0 = all)
--batch-size N	Page size (default 500, max 1000)
--help	Show usage

Step 2 — Run the updated pipeline

The re-ingestion itself is out of scope for this document. It is covered by the SS1.3 work and the demo-bootstrap spec. In practice this step is a full corpus sweep via python3 scripts/ingest.py --file urls.txt, driven by a fresh source-document inventory (see §5) and a rebuilt target database (see database-rebuild-runbook.md).

Step 3 — Generate PipelineExtractionResult records during re-ingestion

Every item passing through the pipeline produces a PipelineExtractionResult via lib/extraction/extraction-result.ts (TypeScript) or scripts/kb_pipeline/extraction_result.py (Python). These are parity-tested in __tests__/lib/extraction-result-parity.test.ts and scripts/tests/test_extraction_result.py.

Write each result as one JSON object per line to data/extractions/reingest-YYYY-MM-DD.jsonl. This artefact is the raw input for any post-hoc per-item quality analysis that the snapshot comparison does not cover (for example, which quality warnings fired during extraction).

Step 4 — Compare dimensions

After the re-ingestion finishes, capture a second snapshot (the “post” snapshot) with the same invocation — typically to data/snapshots/post-reingest-YYYY-MM-DD.jsonl — then compare:

bun run scripts/compare-quality.ts \
  --old data/snapshots/pre-reingest-YYYY-MM-DD.jsonl \
  --new data/snapshots/post-reingest-YYYY-MM-DD.jsonl \
  --output data/reports/re-ingest-report-YYYY-MM-DD.md

Flags, per bun run scripts/compare-quality.ts --help:

Flag	Purpose
--old PATH	Older snapshot JSONL (pre re-ingestion) — required
--new PATH	Newer snapshot JSONL (post re-ingestion) — required
--output PATH	Write markdown report (default stdout)
--help	Show usage

The script runs entirely offline — no Supabase or OpenAI access. Chunk counts are read from the snapshot’s chunk_count field.

Quality dimensions computed (as shipped in WP1b — these supersede the original Plan D spec table):

#	Dimension	Metric	Computation	Threshold
1	Structural fidelity	Heading-count ratio	new.heading_count / old.heading_count, counted by matching ^#{1,6}\s+ in the stored markdown	≥ 90% of items at ratio ≥ 0.9
2	Content completeness	Character-count ratio	new.content_length / old.content_length	Median 0.95–1.10
3	Embedding stability	Cosine similarity	cosineSimilarity(old.embedding, new.embedding)	Median > 0.95, none < 0.90
4	Classification stability	Primary-domain match	old.primary_domain === new.primary_domain	≥ 95% match
5	Entity extraction stability	Jaccard similarity of canonical names	jaccard(Set(old.canonical_names), Set(new.canonical_names))	Mean > 0.90
6	Coverage equivalence	Per-domain item counts	Compare distributions; no domain loses > 10% of its items	No domain at ≤ −10%
7	Chunk quality	Chunks per document	Compare chunk_count across buckets (articles 3–20 avg; question_answer 1)	Within expected range
8 (supplementary)	Body-text completeness	Word-count ratio	new.word_count / old.word_count	≥ 90% of items at ratio ≥ 0.9

Reminder — what changed relative to the original Plan D spec: Dimension 1 in the plan table used a word-count proxy for structural fidelity; the shipped script uses the real markdown-heading count. Dimension 5 in the plan used an entity-count proxy; the shipped script uses Jaccard similarity on the set of canonical names per item. The word-count ratio is still reported, but as Dimension 8 (supplementary), not as the primary structural-fidelity signal. Operators reading the Plan D text should trust the table above.

Step 5 — Produce the quality report

compare-quality.ts writes a self-contained markdown report including:

• a header with the snapshot paths and the paired/unpaired item counts;
• a summary table of the dimensions above, each with status (PASS / FAIL / WARN / N/A);
• an outliers section listing items that failed any individual threshold;
• a per-content-type breakdown (avg similarity, avg char ratio, chunk counts per bucket).

Paste the report verbatim into the session handoff so the decision to proceed with (or revert) the re-ingestion is traceable.

Step 6 — Human review

Before signing off the re-ingestion, the product owner visually compares 10 sampled items end-to-end. Sample composition:

• 2 articles — heading structure preserved, paragraphs intact
• 2 PDFs — page boundaries visible, tables rendered
• 2 Q&A pairs — question / answer separation maintained
• 2 blogs — formatting preserved, code blocks intact
• 2 policy documents — numbered lists and regulatory references intact

For each item compare the old rendered output (plain text via ContentRenderer) with the new rendered output (markdown via ContentRenderer). Capture screenshots of any visual regressions and log them against the session before the decision to proceed.

Bonus step — Embedding smoke test

Run this before committing to the full re-ingestion. It validates that the extraction-format change has not degraded embedding quality on a small, representative sample.

bun run scripts/embedding-smoke-test.ts                      # default 20-item run
bun run scripts/embedding-smoke-test.ts --dry-run            # select items only
bun run scripts/embedding-smoke-test.ts --limit 5            # smaller sample
bun run scripts/embedding-smoke-test.ts --content-type pdf   # single bucket
bun run scripts/embedding-smoke-test.ts --output results.jsonl
bun run scripts/embedding-smoke-test.ts --verbose

Pass criteria (spec SS6.2):

• Median cosine similarity > 0.95.
• No individual item below 0.90.

Known caveat — Python-ingested items. Items originally ingested via the Python pipeline (roughly 400 items at the time of writing) had their embeddings built from only the first 1,500 characters of content, because the Python build_embedding_text helper historically truncated at that limit. As of Plan D Divergence 1 (closed in S168 WP1c), the Python constant is raised to MAX_EMBEDDING_CHARS = 24_000, aligned with the TypeScript classify path (lib/ai/embed.ts). The smoke test will therefore see low similarity by design on Python-ingested items — on average the new embedding is a better representation of the full document (edge cases with unusual extraction behaviour can still regress for a specific item). Flag these items separately, not as failures.

Cost is roughly 20 items × 1 embedding call — well under $0.01 at current OpenAI rates.

---

4. Pipeline divergence assessment

As of S168 WP1c, five divergences are tracked between the TypeScript classify path (lib/ai/classify.ts around line 1133), the MCP create path (lib/mcp/tools/content.ts around line 337), and the Python pipeline (scripts/kb_pipeline/embed.py).

#	Divergence	Severity	Status
1	Content truncation ceiling	High	CLOSED (S168 WP1c) — Python now uses MAX_EMBEDDING_CHARS = 24_000, matching TypeScript
2	Summary inclusion in embedding input	Medium	Documented, kept intentionally
3	Transcript / content_type == "other" special case	Low	Documented, not fixed (niche path)
4	Extraction library differences	Low	Documented, acceptable (quality compared via embedding smoke test)
5	MCP create tool uses 5,000-char truncation	Medium	Open — backlog follow-up

Divergence 1 — Content truncation ceiling (closed)

The Python build_embedding_text previously truncated content at 1,500 characters — roughly the first two or three paragraphs of a typical policy document. The TypeScript classify path uses 24,000 characters (see MAX_EMBEDDING_CHARS in lib/ai/embed.ts). As of S168 WP1c the Python helper defines its own module-level MAX_EMBEDDING_CHARS = 24_000 constant with a docstring cross-referencing the TypeScript file as the single source of truth.

Divergence 2 — Summary inclusion (documented, kept)

Path	Title	Summary	Content
Python pipeline	extracted title	included	up to 24,000 chars
TS classify	AI-generated suggested_title	not included	up to 24,000 chars
TS MCP create	user-provided args.title	not included	up to 5,000 chars (see Divergence 5)

The Python path’s inclusion of the AI-generated summary is retained deliberately: the summary is a compact, semantically-rich signal at a negligible token cost. Removing it would degrade existing Python-ingested item embeddings without a corresponding quality gain. Going the other way — adding the summary to the TypeScript paths — is an optional follow-up, not a defect.

Divergence 3 — Transcript handling (documented, not fixed)

The Python path has a special branch for content_type == "other" (the legacy transcript type) that uses chapter titles from metadata.chapters as a topic outline instead of the truncated transcript body. The TypeScript paths have no equivalent. This is a niche path — transcripts are rarely ingested — and the branch is documented in the Python helper’s docstring. Keep as-is.

Divergence 4 — Extraction library differences (documented, acceptable)

Source	Python	TypeScript
HTML (web)	trafilatura (plain text), with jina_reader fallback	Readability (plain text)
PDF	pdfplumber	unpdf
DOCX	python-docx	mammoth

After Plan B (markdown canonical), both pipelines emit markdown, but via different libraries. The embedding smoke test measures the output quality end-to-end, so differences in the intermediate extraction library are acceptable provided the smoke test passes on both sides.

Divergence 5 — MCP create truncation (open)

lib/mcp/tools/content.ts (around line 337) builds the embedding input as args.title + ' ' + args.content.slice(0, 5000). This is stricter than the 24,000-char ceiling on the classify path, and stricter than the Python path now uses. Effect is limited to items created via the MCP create_content_item tool. Tracked as backlog item MCP-EMBED-1 (docs/reference/product-backlog.md §6): align to MAX_EMBEDDING_CHARS once the Plan B markdown canonical format is confirmed stable for MCP-created items.

---

5. DOCX pre-processing checklist

Applied before ingesting any DOCX batch. Source: spec SS7.1.

1. Track Changes detection. Run scripts/docx_utils.has_tracked_changes(path). If True and the Python pipeline is used, scripts/docx_utils.open_document_safe(path) resolves them automatically via pandoc (--track-changes=accept), provided pandoc is installed (brew install pandoc). If True and the TypeScript pipeline is used, mammoth accepts all changes during HTML conversion without further action. For detailed stats, scripts/docx_utils.get_track_changes_stats(path) returns {has_changes, insertion_count, deletion_count}.

2. Embedded objects. Images, charts, SmartArt, and OLE objects are lost during text extraction. If the DOCX contains them, log a quality warning against the item. mammoth emits warnings for unsupported content types — capture them into the extraction result.

3. Character encoding. DOCX is XML-based and UTF-8 by default, so encoding issues are rare. Watch for documents that started life as legacy .doc files, and for documents containing pasted content from non-UTF-8 sources.

4. Password protection. Both mammoth and python-docx fail silently on password-protected documents. Before ingestion, attempt to open each file with open_document_safe(); if the call raises, flag the file as protected and exclude it from the batch.

5. Filename standardisation. Lowercase, replace spaces with hyphens, remove special characters. Store the original filename in metadata for traceability. Example: "Q&A Response - Final Draft (v2).docx" → "qa-response-final-draft-v2.docx".

---

6. Source document inventory template

Filled in manually before each batch ingestion. Source: spec SS7.2.

# Source Document Inventory: [Batch Name]

Date: DD/MM/YYYY Prepared by: [Name]

## Summary

- Total documents: N
- Formats: X DOCX, Y PDF, Z markdown
- Track Changes detected: N files
- Estimated ingestion time: ~Xm (at 1.5s rate limit per item)

## Inventory

| #   | Filename     | Format   | Size (KB) | Track Changes      | Tables | Content Type (est.) | Notes                                  |
| --- | ------------ | -------- | --------- | ------------------ | ------ | ------------------- | -------------------------------------- |
| 1   | example.docx | DOCX     | 245       | Yes (3 ins, 1 del) | 2      | article             | Track Changes auto-resolved by mammoth |
| 2   | policy.pdf   | PDF      | 1,200     | N/A                | 5      | article             | 12 pages, table-heavy                  |
| 3   | faq.md       | Markdown | 8         | N/A                | 0      | question_answer     | Already markdown                       |

## Pre-processing actions taken

- [ ] All Track Changes resolved (mammoth auto-resolves for TS path; pandoc for
      Python path)
- [ ] Password-protected files identified and excluded
- [ ] Embedded objects noted in quality warnings
- [ ] Filenames standardised
- [ ] Content types pre-assigned where obvious from filename or folder structure

## Quality baseline

- Items with existing embeddings: N (for smoke-test comparison)
- Items being re-ingested vs new: N re-ingest, M new

Per spec SS7 resolved question 7, automation tooling around the inventory is deferred until after the first re-ingestion produces enough operational data to decide what is worth automating.

---

7. Cross-references

• two-stage-re-ingestion-runbook.md — operational execution guide for the Phew blank-DB re-ingestion (Stages 0-2). The decision gate (Step 20) invokes this protocol.
• blank-db-restore-matrix.md — 25-step FK-ordered restore sequence; Step 20 is the quality gate that uses this protocol.
• database-rebuild-runbook.md — end-to-end rebuild sequencing; §11 Cleanup points at this protocol as the quality gate before deleting the old project.
• .planning/.archive/.specs/content-format-standardisation-spec.md — SS6 (quality measurement) and SS7 (pre-processing) are the spec authorities behind this document.
• .planning/.archive/.specs/plan-d-quality-documentation.md — the source plan for tasks D1–D8; this file implements D4, D5 (documentation portion), D6, and D7.
• docs/audits/two-pass-cost-quality-measurement.md — two-pass cost vs quality measurement (Session 168 WP2) that feeds into the re-ingestion decision.
• Scripts:
  • scripts/snapshot-content-state.ts — Step 1
  • scripts/compare-quality.ts — Step 4
  • scripts/embedding-smoke-test.ts — bonus step
  • scripts/kb_pipeline/embed.py — Python embedding helper (closed Divergence 1 in S168 WP1c)
  • scripts/docx_utils.py — Track Changes detection and pandoc resolution
• Factories: lib/extraction/extraction-result.ts (TypeScript), scripts/kb_pipeline/extraction_result.py (Python).