{56.18} SPIKE — AST heading-population go/no-go for the chunking stage

{56.18} SPIKE — AST heading-population go/no-go for the chunking stage

Status: RESEARCH SPIKE for ratification. Evidence-backed go/no-go on populating the four heading-derived content_chunks columns (heading_text / heading_level / heading_path / parent_chunk_id) from the cocoindex chunking stage via an AST (tree-sitter) route. Surfaced by Lane A (id56-heading, OQ oq-fedd4fc673da6349) under the ratified (a) keep-nullable disposition (heading-cols-disposition-RESEARCH.md; Liam S286, OQ oq-c1a83ca097293fb6). This spike does NOT change that disposition.

Harness: scripts/spikes/ast-heading-population-eval.py (non-production, no network, no credentials — run PYTHONUNBUFFERED=1 python3 scripts/spikes/ast-heading-population-eval.py).

---

Question

Does cocoindex 1.0.3 RecursiveSplitter(language='markdown').split surface heading-path-mappable node metadata that the chunking stage (scripts/cocoindex_pipeline/flow.py:1965-2026, currently a bare RecursiveSplitter() with NO language= arg and the four heading columns deliberately omitted) could use to populate heading_text / heading_level / heading_path / parent_chunk_id — and if so, is the retrieval/structural benefit worth the cost?

Context: the {56.5} spike only tuned byte budgets (Variant-B 2000/200/1000) — language= was never exercised (git pickaxe git log -S 'language=' -- flow.py empty across all branches, per the {56.17} handoff packet).

Method

Staged protocol, stop-early on no-go evidence:

1. Stage 1 — empirical metadata check. Introspect the installed package (pip show cocoindex → 1.0.3 confirmed; /Users/…/site-packages/cocoindex/): split signature, the Chunk resource dataclass, the raw PyO3 chunk attribute surface beneath the Python wrapper, the _convert_chunk source, strings probe of the native engine (_internal/core.abi3.so), and behavioural batteries (language=None vs 'markdown' vs unknown) on synthetic adversarial markdown and real repo markdown at Variant-B budgets.
2. Stage 2 — prototype (reached, because Stage 1 returned a viable-with-caveat verdict): minimal mapping from splitter output to the four columns, mirroring the legacy chunker’s semantics (scripts/kb_pipeline/chunk.py:78-107,266-281).
3. Stage 3 — eval (reached): structural proxy substituted for the credentialed recall@k eval — substitution justified in §Stage-3 below.

---

Stage-1 evidence

1a. API surface — verbatim captures

cocoindex version : 1.0.3
split signature   : (self, text: str, chunk_size: int, *, min_chunk_size: int | None = None,
                     chunk_overlap: int | None = None, language: str | None = None)
                     -> list[cocoindex.resources.chunk.Chunk]
Chunk fields      : ['text', 'start', 'end']
raw PyO3 chunk attrs (8): ['end_byte', 'end_char_offset', 'end_column', 'end_line',
                           'start_byte', 'start_char_offset', 'start_column', 'start_line']
detect_code_language('x.md') -> markdown
unknown language: no error, 1 chunks (silent fallback)

The public Chunk dataclass (cocoindex/resources/chunk.py) carries exactly text, start, end (each a TextPosition with byte_offset / char_offset / line / column). The Python wrapper’s _convert_chunk (cocoindex/ops/text.py) maps every attribute the Rust engine exposes — the raw PyO3 chunk has only the 8 offset/position attributes above. There is no node kind, no heading ancestry, no AST handle anywhere on the output surface, in any form. Nothing is dropped by the wrapper; the engine simply does not surface it.

Two further API facts:

• language='markdown' IS a recognised built-in (detect_code_language('x.md') → 'markdown'; aliases .md, md, .markdown, etc. visible in the engine binary).
• An unrecognised language silently falls back to plain-text splitting — no error, no introspectable signal. There is no API to ask “was tree-sitter actually used?“.

1b. The engine DOES parse markdown with tree-sitter — internally only

strings probe of _internal/core.abi3.so (47 MB): embeds tree-sitter 0.25.10 and the tree-sitter-md grammar’s node-kind strings — atx_heading, setext_heading, heading_content, setext_h1_underline, fenced_code_block, indented_code_block, pipe_table, block_quote, minus_metadata — plus the markdown extension alias table. So language='markdown' engages a real tree-sitter markdown parse for boundary placement only; the parse tree is consumed inside Rust and never crosses the FFI.

1c. Behavioural batteries — what language='markdown' actually changes

Synthetic adversarial docs at Variant-B budgets (2000/200/1000 bytes):

Doc shape	lang=None	lang='markdown'	Identical?
Well-sized sections (~1.1-1.6 kB each, nested H1/H2/H3)	6 chunks, 6/6 heading-at-start, 5 heading occurrences mid-chunk (overlap bleeds across section boundaries)	6 chunks, 6/6 heading-at-start, 0 mid-chunk (overlap clamped at section boundary, e.g. chunk ends at 2573 just before the heading at 2575)	No — ends differ
10 tiny sections (~160 B each, merge pressure)	1 chunk, 9 headings mid-chunk	1 chunk, 9 headings mid-chunk	Yes
One oversized section (6.2 kB body)	4 chunks, 1 heading mid-chunk	4 chunks, 1 heading mid-chunk	Yes
Mixed sizes, nested	5 chunks, 5/6 headings mid-chunk	5 chunks, 5/6 mid-chunk	Yes
Fake # headings inside a code fence	no split inside fence	no split inside fence	Yes

Real repo markdown at Variant-B budgets (heading-alignment improves markedly):

Doc	mode	chunks	heading-at-chunk-start	heading occurrences mid-chunk
docs/testing/uat/test-bid-resources.md (51.5 kB, 40 headings)	None	34	9/40	37
	markdown	35	22/40	18
docs/reference/state-of-the-product.md (168 kB, 43 headings)	None	114	9/43	39
	markdown	118	23/43	20
docs/runbooks/ci.md (47.5 kB, 35 headings)	None	31	8/35	35
	markdown	32	21/35	15

Stage-1 verdict

No heading-mappable NODE metadata exists in any form — the strict premise of “AST heading-population” (tree-sitter node metadata → columns) fails. However, the protocol’s “any form” test is not fully empty: offsets ARE surfaced, and offsets + an independent source-side heading index make the four columns deterministically derivable by post-processing. And language='markdown' materially improves how truthful such derived values would be (heading-aligned starts roughly ×2.5, mid-chunk headings roughly halved, on real markdown). So Stages 2-3 were run rather than stopping — the go/no-go then turns on cost vs measured benefit, not on raw possibility.

---

Stage-2 prototype

Full runnable prototype in scripts/spikes/ast-heading-population-eval.py (build_heading_index + derive_heading_columns). Core mapping (~60 lines, pure post-processing inside the existing @coco.fn — no new cocoindex API needed):

@dataclass(frozen=True)
class Heading:
    offset: int   # char offset of the heading line start
    level: int
    text: str

def build_heading_index(text: str) -> list[Heading]:
    """ATX headings with char offsets, fence-aware (setext: see Open questions)."""
    headings, in_fence, pos = [], False, 0
    for line in text.splitlines(keepends=True):
        stripped = line.strip()
        if stripped.startswith("```") or stripped.startswith("~~~"):
            in_fence = not in_fence
        elif not in_fence:
            m = re.match(r"^(#{1,6})\s+(\S.*)$", line)
            if m:
                headings.append(Heading(pos, len(m.group(1)), m.group(2).strip()))
        pos += len(line)
    return headings

def derive_heading_columns(chunk_starts, headings):
    """Per chunk: governing heading = deepest heading at-or-before chunk start;
    heading_path = ancestor stack incl. governing; parent = first chunk of the
    nearest ancestor (level < governing) heading's section."""
    ...

Demo output on the UK-procurement contract fixture (__tests__/fixtures/cocoindex-chunking/long-terms.md, markdown mode):

chunk 0: [    0, 1987) H1 'Contract Terms and Conditions' | path: Contract Terms and Conditions | parent_position: None
chunk 1: [ 1989, 3800) H2 'Termination and Exit' | path: Contract Terms and Conditions > Termination and Exit | parent_position: 0
chunk 2: [ 3802, 5119) H2 'Intellectual Property' | path: Contract Terms and Conditions > Intellectual Property | parent_position: 0

Notable simplification vs legacy: parent_chunk_id needs no second-pass UPDATE (the legacy writer PATCHed parents after insert, scripts/kb_pipeline/chunk.py:266-281) — under the flow’s deterministic id scheme it is simply uuid5(_KH_PIPELINE_DOC_NS, f"chunk:{rel_path}:{parent_position}"), computable inline.

Stage-3 eval — design, substitution justification, results

Substitution: structural proxy instead of credentialed recall@k. Justification:

1. The {56.5} recall@k harness’s corpus is extracted text from binary tenders (docx/pdf/xls). Empirically (table below), the docx and pdf extractions contain zero markdown headings — language='markdown' has no traction and the heading columns would be NULL on that corpus shape under ANY disposition. A recall@k re-run there cannot measure the heading question.
2. Constructing a NEW credentialed ground truth (query → relevant section) over a markdown procurement corpus, plus live embeddings, is disproportionate to a metadata-population decision: the columns feed display/breadcrumb consumers (formatChunkSearchResults, formatContentItemChunks), not the vector-ranking path.
3. The structural proxy measures exactly the property that determines whether populated values would be truthful: do chunks respect section boundaries, and what fraction of a chunk’s content actually belongs to the section its derived heading_path claims?

Metrics (per doc × mode, Variant-B budgets 2000/200/1000): chunk count; heading-aligned chunk-start rate; section purity (chunks with zero mid-chunk headings); truthfulness — fraction of chunk chars lying inside the section span of the derived governing heading. Corpus: two shapes — (a) UK-procurement markdown (long-terms.md contract fixture, short-clause.md, test-bid-resources.md) and (b) the {56.5} binary tender fixtures via the same extractors (docs/testing/test-data/templates/). Chosen because shape (b) is what the pipeline actually ingests from clients; shape (a) is the only shape where headings exist at all.

Results (verbatim from the harness):

doc                              lang        chars hdgs chunks hdg-aligned      pure truthful
---------------------------------------------------------------------------------------------
[md] long-terms.md               None         5120    9      3         3/9       0/3    50.5%
[md] long-terms.md               markdown     5120    9      3         3/9       0/3    64.8%
[md] short-clause.md             None          756    1      1         1/1       1/1   100.0%
[md] short-clause.md             markdown      756    1      1         1/1       1/1   100.0%
[md] test-bid-resources.md       None        51528   40     34        9/40     14/34    78.5%
[md] test-bid-resources.md       markdown    51528   40     35       22/40     21/35    89.0%
[extracted] ITT Services.docx    None       139199    0     90         n/a     90/90   100.0%
[extracted] ITT Services.docx    markdown   139199    0     90         n/a     90/90   100.0%
[extracted] SQ PPN 03/24.pdf     None       124815    0     85         n/a     85/85   100.0%
[extracted] SQ PPN 03/24.pdf     markdown   124815    0     83         n/a     83/83   100.0%
[extracted] ITT Eval Matrix.xls  None         7372   13      4        2/13       0/4    38.8%
[extracted] ITT Eval Matrix.xls  markdown     7372   13      4        3/13       1/4    43.4%

Readings:

• Shape (b) — the real ingest corpus — has nothing to populate. The docx and pdf extractions surface 0 markdown headings (the “100% pure / 100% truthful” rows are trivially so: every chunk is preamble). The xls rows’ 13 “headings” are the extractor’s own synthetic ## Sheet: markers, and even those map poorly (≤43% truthful). On the client tender corpus, populated heading columns would remain NULL regardless of GO.
• Shape (a) — markdown sources — population is possible but capped. Budget-driven splitting merges small sections (min_chunk_size=1000 vs typical ~500-char contract clauses: long-terms.md gets 3 chunks across 9 sections, 0 pure chunks). Best case with language='markdown': 89% truthful. Worst case as-built (bare splitter, lang=None column): populating WITHOUT also passing language='markdown' would label half the content of a contract-style doc with the wrong section (50.5%).
• language='markdown' is a genuine boundary-quality improvement on markdown sources (independent of whether columns are populated) — but it changes chunk boundaries, hence chunk text/embeddings, and the Variant-B ratification ({56.5}) was made on bare-split behaviour.

---

Recommendation: NO-GO (for v1; keep-nullable disposition unchanged)

The premise of “AST heading-population” fails at Stage 1: cocoindex 1.0.3 surfaces no AST node metadata — population would be a self-built source-scan mapping (substantively reviving the retired legacy chunker’s heading logic, scripts/kb_pipeline/chunk.py, layered on top of budget-split boundaries). The measured payoff does not justify the cost:

1. Zero coverage on the primary ingest corpus. Extracted docx/pdf tenders carry no markdown headings — the columns stay NULL for those items under GO anyway.
2. Capped truthfulness on markdown sources. 65-89% of chunk content correctly labelled at Variant-B budgets even with language='markdown'; mislabelled breadcrumbs in MCP search results are arguably worse than the current honest (preamble) / (document root) rendering (consumers are already NULL-tolerant end-to-end, per heading-cols-disposition-RESEARCH.md §3).
3. The keep-nullable disposition (a) anticipated exactly this: columns stay, NULL is rendered gracefully, and the door stays open if a future heading-aware splitter (or richer cocoindex output surface) lands upstream.

What GO would cost (recorded so the ratification is informed)

• flow.py chunking stage (scripts/cocoindex_pipeline/flow.py:1965-2026): pass language= (markdown only — needs a source-format signal at the split call), add the ~60-line heading-index + mapping helpers, add the four keys to the declare_row dict, and add the four columns to CONTENT_CHUNKS_SCHEMA (flow.py:1234-1257).
• No schema migration — that is the point of disposition (a): all four columns exist with the right types; heading_path’s '{}' default simply gets real values.
• Tests: scripts/tests/test_cocoindex_chunking.py:16-19,299-311 currently ASSERTS the four keys are omitted — rewrite plus new mapping tests (setext, fences, preamble, parent linking).
• Backfill posture: re-ingest-driven only. ingest_file is memo=True; whether a splitter-config change re-fingerprints unchanged-bytes documents is unverified (open question below) — worst case the corpus stays NULL until content changes or a forced re-ingest.
• Ratification debt: Variant-B ({56.5}) was ratified on bare-split behaviour; language='markdown' shifts boundaries (±1-4 chunks per doc observed), so chunk ids/text/embeddings change on next re-ingest — strictly a re-ratification of the chunk config.
• Hardening: the prototype indexes ATX headings only; production needs setext (===/--- underlines) and ~~~-fence parity with tree-sitter-md’s view, or derived values silently diverge from the boundaries the engine chose.

Separable rider (NOT recommended by default, listed for completeness)

Passing language='markdown' for markdown-sourced items WITHOUT populating any columns is a pure boundary-quality improvement (mid-chunk headings roughly halved). It still carries the flow.py touch, the {56.5} re-ratification debt, and the re-ingest churn — so the default remains: leave the stage as built.

Open questions

1. Memo interaction: does changing splitter arguments (language=) invalidate the memo=True fingerprint for unchanged-bytes documents, or would a backfill require a forced re-ingest? (Determines GO’s backfill posture; unverified — needs a check of _internal/memo_fingerprint.py semantics vs the flow’s memo key.)
2. Source-format signal: the chunking stage currently has no clean “this content_text is native markdown vs extracted text” discriminator; a GO would need one (e.g. source file extension threaded into ingest_file) to avoid feeding language='markdown' to extracted prose.
3. Upstream trajectory: if a future cocoindex release surfaces chunk-level node metadata (e.g. heading ancestry per chunk), the cost side collapses — worth a pin-bump check at the next cocoindex upgrade.
4. Prod heading values: heading-cols-disposition-RESEARCH.md §6 verified zero populated heading values on staging (408 chunks); prod remains UNVERIFIED — irrelevant to this NO-GO but still the gating check if option (b) (drop columns) is ever revisited.

Sources

• Installed package: cocoindex==1.0.3 (requirements.txt pin), introspected at ~/Library/Python/3.14/lib/python/site-packages/cocoindex/ — ops/text.py (RecursiveSplitter, _convert_chunk), resources/chunk.py (Chunk, TextPosition), _internal/core.abi3.so (strings probe).
• As-built stage: scripts/cocoindex_pipeline/flow.py:1215-1257 (config + schema), :1965-2026 (chunking stage, bare RecursiveSplitter(), heading keys OMITTED).
• Legacy semantics mirrored: scripts/kb_pipeline/chunk.py:78-107 (heading stack, parent_position), :266-281 (second-pass parent PATCH).
• Disposition + sequencing: heading-cols-disposition-RESEARCH.md; heading-cols-56.17-sequencing-amend-handoff.md.
• Harness + verbatim outputs: scripts/spikes/ast-heading-population-eval.py.