DR-144: The OKF producer adopts ruamel.yaml for frontmatter; the hand-rolled emitter and parser retire

DR-144 — The OKF producer adopts ruamel.yaml for frontmatter

Context

scripts/cocoindex_pipeline/producer/frontmatter.py hand-rolls both the YAML frontmatter emitter and (since id-440) the concept-document parser. The constraint that produced this came from exactly one place — not a spec, not a skill, not an agent-harness instruction, and no prior decision. It is that module’s own docstring, authored in the same id-426/S546 wave that introduced sources::

“Deliberately hand-rolled (no pyyaml dependency): PyYAML is not pinned in requirements.txt (only resolves transitively in this environment), and the frontmatter shape here is a small, fully-controlled subset.”

Re-measured at S548 and confirmed: yaml is absent from requirements.txt and resolves only transitively (requirements.lock:507, pyyaml==6.0.3); no module under scripts/ imports it; ruamel has no hit in either file and is not importable in the environment — a genuinely new dependency, not a promoted transitive one.

id-440’s agent honoured that docstring correctly under DR-123’s carrier rule (a contemporaneous docstring outranks a later task file’s recollection — id-440’s own Goal asserted the opposite, falsely). The owner has now overturned it at the requirement level, which is the only level that can.

Decision

Adopt ruamel.yaml as a pinned producer dependency, in id-428. The hand-rolled emitter and parser retire in its favour. frontmatter.py’s docstring is rewritten, not merely contradicted in code.

id-428 owns the migration because both changes rewrite the same module; splitting them would mean writing a throwaway intermediate state. The library migration sequences first, so a verified:/status: regression is never ambiguous between the two causes.

Alternatives Considered

PyYAML. Never a real candidate, and the reason is structural rather than preferential: safe_load/safe_dump normalises the generated: { by, at } flow mapping into block form and re-quotes scalars, so it cannot satisfy id-440’s AC-1 byte-faithful round-trip by construction. Only a round-trip-preserving library can.

Keep hand-rolling and fix the known bug. Defensible — the emitted shape is a small controlled subset, the producer is its sole writer, and the deploy surface (the IONOS VPS pipeline host) gains a runtime dependency. Rejected by the owner in favour of retiring the maintenance surface wholesale rather than patching it once more.

Consequences

The central hazard, stated because it is easy to assume away: ruamel’s round-trip mode preserves what it parsed. frontmatter.py emits fresh, so round-trip fidelity is not automatic on the write path and must be demonstrated, not inherited from the library’s reputation.

Two guarantees are load-bearing, both won by measurement in id-440, and neither may be surrendered silently:

• AC-1 — a byte-faithful capture → reapply cycle over real emit_concept_frontmatter output. Re-prove it against ruamel using id-440’s existing 14 regression tests before deleting any hand-rolled path.
• AC-2 — _parse_concept_doc re-renders what it parsed and compares against its own source lines, raising FrontmatterShapeError on mismatch. That guarantee is shape-general: it holds for shapes nobody enumerated. A library swap must keep the loud-refusal property; degrading it back to silent truncation is the defect id-440 fixed.

The concrete bug this retires: _yaml_escape escapes \ and " but not newlines, so a title or description containing a raw newline still emits a broken multi-line double-quoted scalar. Surfaced by id-440 and handed to id-428. It is the concrete instance of the general cost of hand-rolling, and the migration should be measured against it.

Scope boundary — this ruling does not reach the consumer. lib/okf/ parses OKF documents in TypeScript and is untouched by a Python dependency choice. Whether the TS side should take a YAML library is a separate, unasked question.