Skip to content
Canonical

PLAN — {42.4} Deploy pullmd Cloud Run Service + retire HTML extraction cascade

PLAN — {42.4} Deploy pullmd Cloud Run Service + retire HTML extraction cascade

Section titled “PLAN — {42.4} Deploy pullmd Cloud Run Service + retire HTML extraction cascade”

Task: ID-42 — Deploy pullmd Cloud Run Service + retire current HTML extraction cascade. Spec-chain artefact: {42.4} PLAN (fourth of {42.1}→{42.2}→{42.3}→{42.4}). Date: 25/05/2026 (S265 / ID-42 spec wave). Author: Task Planner (Opus 4.7, 1M context) — fresh instance per Q-PLANNER-2; NOT the {42.1}/{42.2}/{42.3} author. Fresh decomposition pass over both ratified predecessors. Predecessors (read in full): docs/specs/id-42-pullmd-deploy/TECH.md (18 proposed changes, WP-A…WP-G); docs/specs/id-42-pullmd-deploy/PRODUCT.md (18 invariants Inv-1…18); docs/specs/id-42-pullmd-deploy/RESEARCH.md (empirical baseline). Status: PLAN draft. Returns the TM-shape Subtask records {42.5…42.13} to the Orchestrator for append to docs/reference/task-list.json under Task ID-42. The Planner does NOT edit task-list.json.

UK English throughout. This document records the decomposition rationale and — its load-bearing purpose — the dependency-gate structure that encodes Inv-10 (no-deletion-before-proof) and the cross-Task ID-28.20 cocoindex Stage-6 write gate.

1. Decomposition summary

Section titled “1. Decomposition summary”

Nine implementation Subtasks, {42.5}…{42.13}, mapping the seven TECH work-packages (WP-A…WP-G) onto S/M-sized, independently-dispatchable, acceptance-verifiable units. Well within the 25-Subtask soft ceiling (§3.4 / A7) — no Task split warranted.

SubtaskWPTitle (short)SizeGate class
42.5A+BAdapter rewrite + test rewrite + httpx pinMLand-now
42.6C (1/2)pullmd manifests + Secret Manager bootstrapMLand-now (modulo GCP access)
42.7C (2/2)Workflow wiring + deploy + smoke-verify + URL replaceMLand-now (modulo GCP access); dep 42.6
42.8DProvenance migration + types regen + CV updateMLand-now
42.9EWrite-site wiring (provenance onto source_documents)S–MID-28.20-gated
42.10FEnd-to-end HTML ingest proof (Inv-7/8/9)MID-28.20-gated (verification gate-opener)
42.11G (1/3)Surface A retirement + ExtractionResult union extendMVerification-gated (Inv-10)
42.12G (2/3)Surface B retirement (HTML branch; keep PDF)SVerification-gated (Inv-10)
42.13G (3/3)npm-dep + workflow-secret removal + scoped grep gateSVerification-gated (Inv-10); deps 42.11+42.12

The existing spec-authoring Subtasks {42.1}-{42.4} stay as-is; implementation Subtasks begin at id 5.

2. Three slicing decisions (and why)

Section titled “2. Three slicing decisions (and why)”

  1. WP-A folds WP-B and its own test rewrite into one Subtask (42.5). The TECH says WP-B (the one-line httpx pin) “could fold into WP-A”, and the existing test_cocoindex_adapters.py tests codify the OLD contract (POST /extract + .json()["markdown"], verified at lines 398-458) — they break the build the instant the adapter changes. Splitting the adapter from its own tests would leave the repo in a non-building state between Subtasks, violating “each Subtask leaves the system working”. One TDD-coupled M slice.

  2. WP-C splits into two along the artefact↔CI seam (42.6 → 42.7). WP-C is the only L-sized work-package (manifests + workflow wiring + secrets + smoke-verify + URL replace) and the only one carrying a GCP/WIF access-wall risk. 42.6 produces the static artefacts (the two manifests) plus the out-of-band Secret Manager bootstrap (token + admin creds); 42.7 wires the workflow, runs the deploy, smoke-verifies, and REPLACES the PULLMD_SERVICE_URL placeholder (which requires the Service to exist first). Clean dependency: 42.7 consumes 42.6. The access risk is flagged in BOTH (the Executor must raise via OQ-pending.md if it hits a project/WIF wall — it cannot self-serve GCP admin).

  3. WP-G splits into three by subsystem + a clean-up gate (42.11, 42.12, 42.13). Surface A (sector-intelligence) and Surface B (URL-ingest) are independent subsystems; the npm-dep removal + workflow-secret removal + the scoped grep gate can only run once BOTH surfaces stop importing the libraries. 42.13 therefore depends on 42.11 AND 42.12. Inv-10 forbids any single commit that both deletes a legacy surface and first introduces the pullmd path — the split keeps deletion strictly downstream of the 42.10 proof.

3. Dependency graph (the load-bearing structure)

Section titled “3. Dependency graph (the load-bearing structure)”

3.1 Within-Task (sibling-only numeric deps — A6 §3.3)

Section titled “3.1 Within-Task (sibling-only numeric deps — A6 §3.3)”
                LAND-NOW (no gate, dispatch immediately)
                ┌──────────────────────────────────────────────┐
   42.5 (A+B adapter+tests+httpx)        42.8 (D migration)
        │                                     │
        │   42.6 (C manifests+secrets)        │
        │        │                            │
        │   42.7 (C workflow+deploy+URL) ◄────┘ (none; 42.7 dep 42.6 only)
        │        │
        ▼        ▼
   ═══════════ ID-28.20 GATE (cross-Task — see §3.2) ═══════════
        │        │
   42.9 (E write-site) ── dep 42.5, 42.8 ── + BLOCKED-ON ID-28.20
        │
   42.10 (F end-to-end proof) ── dep 42.7, 42.9 ── + BLOCKED-ON ID-28.20
        │   (this is the Inv-10 gate-opener: proof green in prod)
        ▼
   ═══════════ Inv-10 VERIFICATION GATE (retirement strictly after) ═══════════
        │
   42.11 (G Surface A) ── dep 42.10
   42.12 (G Surface B) ── dep 42.10
        │        │
        └────────┴──► 42.13 (G npm+workflow+grep gate) ── dep 42.10, 42.11, 42.12

Sibling numeric deps (what goes in each record’s dependencies[]):

Subtaskdependencies (sibling ids)Rationale
42.5[]Pure land-now; touches only adapters.py + its tests + requirements.txt.
42.6[]Static manifests + out-of-band secrets; no code dep.
42.7[6]Consumes the 42.6 manifests + secrets; deploys + replaces URL.
42.8[]Pure DDL; independent of all.
42.9[5, 8]Needs the adapter surfacing headers (42.5) + the columns in the DB (42.8).
42.10[7, 9]Needs the deployed Service + real URL (42.7) + the write site (42.9).
42.11[10]Inv-10: retirement strictly after the proof.
42.12[10]Inv-10: retirement strictly after the proof.
42.13[10, 11, 12]npm-dep + grep gate can only pass once both surfaces are clean.

3.2 Cross-Task gate (ID-42 → ID-28.20) — encoded in prose, NOT in dependencies[]

Section titled “3.2 Cross-Task gate (ID-42 → ID-28.20) — encoded in prose, NOT in dependencies[]”

The Orchestrator’s task-list.json uses sibling-only Subtask deps within a Task (§3.3 / A6). The gate that 42.9 / 42.10 / 42.11 / 42.12 / 42.13 wait on is cross-Task: they cannot complete until ID-28.20 (“functional Stage-6 collect/export write path”) lands, because the cocoindex Surface-C row-write (flow.py content_text.bind_target(...) + flow['op_id']) is a documented non-functional # type: ignore placeholder in cocoindex 1.0.3 (TECH R1 / OQ-42.3-A; ID-28.20 details confirm bind_target “is NOT a method on the cocoindex DataSlice in ANY version”).

Per the §3.3 forcing function this is NOT a Task-boundary violation requiring a split — ID-28.20 is a Subtask of a DIFFERENT Task (ID-28), and ID-42 already depends on the cocoindex pipeline at the Task level. The correct encoding is therefore:

  • A prose BLOCKED-ON: ID-28.20 — cocoindex Stage-6 corpus write line at the head of the details of each gated Subtask (42.9, 42.10, 42.11, 42.12, 42.13).
  • A prominent flag to the parent O-of-O (see the dispatch report) so it can sequence integration: the land-now set (42.5–42.8) dispatches immediately; the gated set waits on ID-28.20 landing.

This is the deliberate fail-safe the TECH’s OQ-42.3-A resolution (A) chose: retirement (WP-G) is blocked by construction until the end-to-end proof (42.10) is genuinely green, so no legacy surface is deleted on an unproven path.

3.3 Inv-10 ordering (no-deletion-before-proof)

Section titled “3.3 Inv-10 ordering (no-deletion-before-proof)”

42.10 (the Inv-7/8/9 end-to-end proof against the DEPLOYED Service, no mock) is the gate-opener. All three retirement Subtasks (42.11/42.12/42.13) depend on 42.10. A Checker can bisect: at any retirement commit’s parent, Inv-7/8/9 already pass, and no single commit both deletes a legacy surface and first introduces the pullmd path.

4. Land-now vs gated (the dispatch buckets)

Section titled “4. Land-now vs gated (the dispatch buckets)”
  • LAND-NOW (dispatch immediately, no ID-28.20 wait): 42.5, 42.6, 42.7, 42.8. Adapter rewrite + httpx pin, pullmd deploy (both halves), migration. These are genuinely independent of the cocoindex write path. 42.6→42.7 is the only internal ordering; 42.5 and 42.8 are fully parallel.
  • ID-28.20-GATED: 42.9, 42.10. Write-site wiring + end-to-end proof. Authored READY (correct columns, correct assertions) so that when ID-28.20 lands the work is thin wiring, not a redesign.
  • VERIFICATION-GATED (after 42.10 green in prod): 42.11, 42.12, 42.13. Retirement.

5. Acceptance-gate items carried (per the ID-42 task record)

Section titled “5. Acceptance-gate items carried (per the ID-42 task record)”
  • Zero-stale-reference grep gate (Inv-15, scoped) — owned by 42.13, including the workflow-YAML probe grep -n FIRECRAWL_API_KEY .github/workflows/cloud-run-deploy.yml → 0.
  • Ops stand-up estimate — docs/plans/phase-0-investigation/0.8.4-pullmd-evaluation.md §10.3 Phase A ≈ 4–6h ops stand-up; cited in 42.6/42.7 details.
  • Provenance (root cause) — the S258 scope-gap discovery (Inv-9 + Inv-7’s implicit dependency on a DEPLOYED pullmd, never stood up; PULLMD_SERVICE_URL left an S258 placeholder) is the root cause this Task closes; cited in 42.7 + 42.10.

6. Risks inherited from TECH (relevant to sequencing)

Section titled “6. Risks inherited from TECH (relevant to sequencing)”
  • R1 / OQ-42.3-A (load-bearing) — cocoindex Stage-6 write path non-functional → the ID-28.20 gate above. Mitigation: 42.9/42.10/42.11/42.12/42.13 carry the BLOCKED-ON prose; the land-now set is unaffected.
  • R3 — --set-secrets is declarative — 42.7 must APPEND PULLMD_API_TOKEN to the existing COCOINDEX_SECRETS string (line 345), never issue a partial update (would drop the other secrets).
  • GCP access wall — 42.6/42.7 may need Liam for project/WIF/Secret-Manager access; both details instruct the Executor to raise via OQ-pending.md rather than improvise.

End of {42.4} PLAN. The TM-shape Subtask records {42.5}…{42.13} are returned to the Orchestrator for append under Task ID-42. The Planner does not edit task-list.json or set any Subtask status beyond the initial pending.