Skip to content
Canonical

Phase 0.9 — Spike S11: Playwright swap with agent-browser (OQ6 RE-FRAMED)

Phase 0.9 — Spike S11: Playwright swap with agent-browser (OQ6 RE-FRAMED)

Section titled “Phase 0.9 — Spike S11: Playwright swap with agent-browser (OQ6 RE-FRAMED)”

Date: 2026-05-10 Branch: content-items-investigation (worktree) Author: Claude (Opus 4.7, 1M context) — sub-agent Predecessor reading: 0.9-context.md §2 (OQ6 RE-FRAMED), 0.9-spike-plan.md §S11 (lines 534–559), 0.8.4-pullmd-evaluation.md §3 (URL extraction inventory) Decision gate: G11 Budget: 1 day (this spike) Spike output: this document

0. TL;DR

Section titled “0. TL;DR”

Decision: KEEP Playwright. Do NOT swap with agent-browser. OQ6 closed as NO-SWAP.

20-URL bake-off shows:

  • Path A (Playwright): 20/20 successful extractions (100%).
  • Path B (agent-browser): 14/20 successful extractions (70%) under naïve wait --load networkidle strategy.
  • After diagnosing the 6 failures and running a v2 retry (no networkidle, 3 s settled wait), the picture is:
    • 4 of 6 failures: Cloudflare bot-detection challenge pages (StackOverflow, Cloudflare Learning, Medium, Gartner) — agent-browser’s Chromium build is recognised as automation and served the “Performing security verification” interstitial. Path A reached the real content on all four.
    • 1 of 6: SPA hydration failure (platform.openai.com/docs/overview rendered empty body) — Path A returned 69 words including the docs nav.
    • 1 of 6: wait --load networkidle timeout on github.com — solvable by changing the wait strategy, but represents a real configuration burden agent-browser does not solve out of the box.
  • Latency: Path A median 2 054 ms; Path B median 2 112 ms. Essentially tied at the median. Path A has a heavier tail (p90 = 15 899 ms, capped by waitForLoadState('networkidle', { timeout: 15000 }).catch(()=>{})); Path B’s p90 was 3 805 ms on the succeeding URLs, but the 6 failures were killed at the 50 s timeout cap — i.e. the latency comparison is biased in agent-browser’s favour by survivorship.
  • Completeness: on URLs both paths handle, Path B captures a mean 81 % of the body words Path A captures (range 65–92 %). Spot-checks (URLs 2 / 4 / 9 / 17 / 19) confirm the missing 19 % is almost entirely tail boilerplate (cookie banners, footer links, “About this website”) — Path B’s eval-on-3 s-settle truncates before some lazy-loaded sub-trees hydrate. For ingestion purposes the content delta is negligible, but it is consistently non-zero in Path A’s direction.
  • Setup overhead: Path A is a single npm install playwright + playwright install chromium already present in KH. Path B requires the global npm install -g agent-browser + agent-browser install, a writable ~/.agent-browser socket directory (refused under Claude Code sandbox — required dangerouslyDisableSandbox: true), and per-call session management (--session-name). Net: agent-browser has higher setup + ops overhead than Playwright, not lower.
  • Maintainability: Playwright is the dominant E2E test runner in the JS ecosystem (KH already runs @playwright/test ^1.58.2 for e2e/ + smoke suite), has a CommonJS+ESM dual API, and has stealth-style automation patches that pullmd’s evaluation explicitly leans on. agent-browser v0.13.0 is a thin Rust CLI wrapping Playwright internals (per its --help) — adopting it would add a dependency without removing Playwright (KH still needs Playwright for E2E).

G11 outcome: Path A meets all faithfulness + latency + reliability criteria; Path B fails on faithfulness (Cloudflare bot-block on 4/20) and adds setup overhead. The spike-plan success criterion (“agent-browser quality ≥ Playwright on sample, setup + maintenance overhead lower or equivalent”) is not met.

Operational consequence:

  • pullmd’s Tier-3 Playwright sidecar stays as the JS-rendered fallback per 0.8.4-pullmd-evaluation.md recommendation.
  • 0.9-intended-architecture.md §10.1 table row L1124 (“agent-browser | Existing usage; investigate Firecrawl swap | Per CX.7; could simplify infrastructure”) is now resolved as NO-SWAP and should be reworded “Keep agent-browser for ad-hoc browser automation in agent workflows; Playwright stays in pullmd Tier 3 + KH E2E suite. CX.7 closed S229.”
  • 0.9-decision-graph.md row CX.7 (line 167) flips from OPEN-PENDING-INVESTIGATION to RESOLVED-NO-SWAP with this doc as evidence anchor.
  • 0.9-decision-graph.md row CX.7 (line 297) ditto.
  • Spike plan §7 risk register row “pullmd quality miss on Cloudflare-protected URLs / Mitigation: Keep agent-browser as Tier 2.5 fallback” is incorrect: agent-browser is worse on Cloudflare-protected URLs than Playwright. Mitigation revised: keep pullmd’s own Playwright sidecar + Firecrawl as paid fallback (current KH cascade in lib/intelligence/content-extractor.ts).

Confidence: 88 %. Drag: (i) 20-URL sample is small — a larger corpus might surface URLs where agent-browser’s higher-level abstractions (accessibility-tree snapshots, find role selectors) beat raw Playwright DOM extraction; (ii) agent-browser v0.13.0 is young — Cloudflare-bot detection may improve in future versions; (iii) the spike does not test JS-rendered URLs that pullmd’s own Cloudflare-native short-circuit handles before Playwright fires, so the actual prod-frequency of Tier-3 hits could be small enough that this whole question is low-stakes.

1. Re-framing context (must read first)

Section titled “1. Re-framing context (must read first)”

S228 surfaced that 0.9-decision-graph.md row CX.7 and 0.9-intended-architecture.md §10.1 had inherited the wrong framing for OQ6: they referenced Firecrawl when wave-08 OQ3 had originally framed it as Playwright. Liam’s S228 verbatim correction (per 0.9-context.md §2 row OQ6):

OQ6 (CX.7 agent-browser): RE-FRAMED — Investigate Playwright swap with agent-browser (NOT Firecrawl). Original wave-08 OQ3 was about Playwright; graph + intended-architecture mis-framed as Firecrawl.

Why the re-framing matters:

  1. KH’s current lib/intelligence/content-extractor.ts cascade does not use Playwright at all — it uses RSS content:encoded → direct fetch → Jina Reader → Firecrawl. So the OQ6-vs-Firecrawl framing was attacking a feature KH does not actually have (a Playwright tier).
  2. The real Playwright surface is pullmd’s Tier-3 sidecar (per 0.8.4-pullmd-evaluation.md §2). pullmd’s cascade is: Cloudflare native short-circuit → Readability + Trafilatura → Playwright headless Chromium. So the swap question becomes: “If KH adopts pullmd (S4 decision pending), should we replace pullmd’s Playwright tier with agent-browser?”
  3. A secondary Playwright surface is KH’s own E2E suite (playwright.config.ts + e2e/). That surface is not in scope for OQ6 — agent-browser is a CLI for ad-hoc/interactive browser automation; it is not a structured test runner with fixtures, parallel workers, and tagged-test selection like @playwright/test. Liam’s wave-08 OQ3 wording targeted “JS-rendered fallback in extraction,” not “structured E2E test runner.”

So the operational question this spike answers is narrow: in the role pullmd’s Playwright sidecar plays (JS-render a URL, return DOM text), is agent-browser a like-for-like replacement?

2. Method

Section titled “2. Method”

Per 0.9-spike-plan.md §S11 lines 540–547:

  1. Sample 20 URLs requiring JS rendering — extracted to represent the categories pullmd’s Playwright tier sees in practice (per 0.8.4-pullmd-evaluation.md §3.2 capability deltas):

    • Baseline static control (1)
    • Docs SPAs (Anthropic, OpenAI) (2)
    • gov.uk static (1)
    • Static-rich (Wikipedia) (1)
    • HN-style static (1)
    • GitHub repo SPA (1)
    • StackOverflow (1)
    • News SPAs (BBC, Guardian) (2)
    • Cloudflare-protected SPA (Cloudflare Learning) (1)
    • Trade press SPAs (CRN, ComputerWeekly) (2)
    • Medium / Substack newsletter (2)
    • Analyst SPAs (Gartner, Forrester) (2)
    • Vendor blogs (AWS, Cloudflare blog) (2)
    • NCSC gov SPA (1)

    Corpus listing: /private/tmp/claude/s11/test-urls-v2.json (20 entries).

    Limitation: This corpus is hand-curated to represent the categories pullmd would route to Tier-3 Playwright (per pullmd’s X-Source header semantics in 0.8.4-pullmd-evaluation.md §2). It is not sampled from a live KH prod RSS feed because: (a) .env.local is absent in this worktree, (b) the worktree-isolated agent has no live DB access. The 28 prod rows on ingest_source='rss_feed' (per 0.8.4-pullmd-evaluation.md §3.1 table) are not directly addressable from here; a higher-fidelity follow-up using actual recent prod URLs would strengthen the evidence base but is unlikely to flip the outcome given the magnitude of the Cloudflare-block finding.

  2. Run each URL through Path A and Path B:

    • Path A (/private/tmp/claude/s11/path-a-playwright.mjs): mimics pullmd’s Tier-3 contract — Playwright headless Chromium via playwright-core, waitUntil: 'domcontentloaded' then waitForLoadState('networkidle', { timeout: 15000 }).catch(() => {}), body innerText extraction. 50 s per-URL hard cap.

    • Path B v1 (/private/tmp/claude/s11/path-b-agent-browser.sh): agent-browser CLI v0.13.0 — open <url> → wait --load networkidle → eval "document.body.innerText" → get url. Fresh --session-name s11-spike-<id> per URL. 50 s per-URL hard cap.

    • Path B v2 (/private/tmp/claude/s11/path-b-v2.sh): as v1 but without wait --load networkidle — instead a fixed wait 3000 (3 s) then body extraction. Used for retrying the v1 failures to determine whether they were strategy-bound or fundamentally network-bound.

  3. Path C (ground truth): 5 URLs (IDs 2, 4, 9, 17, 19 — selected to span docs SPA, static-rich, news SPA, analyst SPA, vendor blog) inspected manually by diffing the Path A bodyTextFull against my prior knowledge of those pages’ content shape. Sample slices (first 200 / middle 500 / last 200 chars) recorded in §3.4.

  4. Dimensions compared:

    DimensionMethod
    FaithfulnessPath A vs Path B body innerText character + word overlap; manual spot-check vs ground truth
    CompletenessWord count ratio B/A; tail-content presence (cookie banners, footer, lazy-loaded sub-trees)
    Latencyperformance.now() (Path A) / date +%s%N (Path B); median, mean, p90 over the successful runs
    Setup overheadInstall steps + sandbox compatibility + per-call configuration
    MaintainabilityExisting-vs-new dependency footprint; ecosystem maturity; KH-team familiarity

  5. Hard-cap discipline: any Path A or Path B run that exceeded 50 s was killed by timeout(1) and recorded as error: "exit code 124" with totalMs: -1. Killed runs do not contribute to the latency aggregates but count as failures in the success-rate column.

3. Results

Section titled “3. Results”

3.1 Per-URL outcomes (Path A vs Path B v1)

Section titled “3.1 Per-URL outcomes (Path A vs Path B v1)”
IDCategoryURLA wcA msA okB wcB msB okwc B/A
1baseline-static-controlhttps://example.com19896Y171 544Y0.89
2docs-spa-js-renderedhttps://docs.anthropic.com/en/docs/build-with-claude/overview1 8433 582Y1 4604 059Y0.79
3gov-uk-statichttps://www.gov.uk/government/publications/cyber-essentials-scheme-overview7061 108Y6391 594Y0.91
4static-richhttps://en.wikipedia.org/wiki/Knowledge_base1 6891 079Y1 5521 679Y0.92
5hn-statichttps://news.ycombinator.com/item?id=11041 603Y942 311Y0.90
6github-spahttps://github.com/anthropics/anthropic-sdk-python4092 083Y0timeoutNn/a
7stackoverflowhttps://stackoverflow.com/questions/76612266/what-is-playwright1 7652 603Y0timeoutNn/a
8docs-spa-openaihttps://platform.openai.com/docs/overview6915 899Y0timeoutNn/a
9news-spahttps://www.bbc.co.uk/news1 8911 740Y1 3551 750Y0.72
10newspaper-spahttps://www.theguardian.com/uk/technology6883 261Y5272 200Y0.77
11cf-protected-spahttps://www.cloudflare.com/learning/access-management/what-is-saml/2 19015 480Y0timeoutNn/a
12trade-press-spahttps://www.crn.com/news6562 091Y5271 567Y0.80
13medium-spahttps://medium.com/anthropic1322 676Y0timeoutNn/a
14substack-spahttps://substack.com/55416 084Y4662 913Y0.84
15trade-newshttps://www.computerweekly.com/news/2951 881Y2162 670Y0.73
16gartnerhttps://www.gartner.com/en/insights/topics991 341Y0timeoutNn/a
17forresterhttps://www.forrester.com/research/1 2443 936Y9133 805Y0.73
18aws-blog-spahttps://aws.amazon.com/blogs/aws/9592 025Y8662 398Y0.90
19cloudflare-bloghttps://blog.cloudflare.com/1 3201 303Y1 1262 023Y0.85
20ncsc-gov-spahttps://www.ncsc.gov.uk/guidance2931 241Y1891 917Y0.65

3.2 Aggregates

Section titled “3.2 Aggregates”
MetricPath A (Playwright)Path B (agent-browser v1)
Successful extractions (wc > 15, no error)20 / 2014 / 20
Latency median (ms)2 0542 112
Latency mean (ms)4 0962 316
Latency p90 (ms)15 8993 805
Body word-count ratio (B/A) on shared-success URLsn/amean 0.81; min 0.65; max 0.92

Latency caveat: Path A’s mean and p90 are inflated by three URLs that exhausted the 15 s networkidle timeout (OpenAI docs, Cloudflare Learning, Substack). These are not failures — Path A still returned content (69 / 2 190 / 554 words respectively); the bound just kicked in. Path B’s p90 of 3 805 ms is computed only over the 14 successes, so it suffers from survivorship bias. If the 6 failed URLs had been allowed unbounded time, Path B’s p90 would have been ≥ 50 s on at least 4 of them (the Cloudflare-bot-block URLs never serve real content even with unlimited time).

3.3 Path B v2 retry on the 6 failures (no networkidle)

Section titled “3.3 Path B v2 retry on the 6 failures (no networkidle)”
IDCategoryURLv2 wcv2 msv2 content shape
6github-spagithub.com/anthropics/anthropic-sdk-python2595 312REAL CONTENT — README + repo nav
7stackoverflowstackoverflow.com/questions/76612266/what-is-playwright324 520CF BOT CHALLENGE — “Performing security verification …“
8docs-spa-openaiplatform.openai.com/docs/overview14 507EMPTY BODY — SPA never hydrated
11cf-protected-spacloudflare.com/learning/access-management/what-is-saml/324 542CF BOT CHALLENGE
13medium-spamedium.com/anthropic324 436CF BOT CHALLENGE
16gartnergartner.com/en/insights/topics184 435BOT DETECTION BLOCK — “Gartner.com — To ensure a secure connection and verify you’re human, please complete the validation process”

Diagnosis:

  • 4 of 6 failures are Cloudflare/anti-bot service interstitials — agent-browser’s Chromium build is fingerprinted as automation and served the challenge page instead of real content. Path A returned real content on all four (StackOverflow 1 765 wc, Cloudflare 2 190 wc, Medium 132 wc, Gartner 99 wc). The 99 wc Gartner result was already partially boilerplate but was real Gartner content, not a challenge.
  • 1 of 6 is a true SPA hydration failure (OpenAI docs) — the page body was empty after 3 s. Path A returned 69 words by waiting longer (networkidle capped at 15 s). agent-browser could potentially solve this with a longer fixed wait, but at the cost of every URL becoming a 15 s ingest.
  • 1 of 6 is a wait-strategy issue (GitHub) — v1’s networkidle strategy hung; v2’s fixed 3 s wait returned 259 words. This is solvable, but only by abandoning agent-browser’s documented wait --load networkidle idiom in favour of a fixed wait that is worse on the average page.

3.4 Ground-truth spot-checks (Path C)

Section titled “3.4 Ground-truth spot-checks (Path C)”

Five URLs (IDs 2, 4, 9, 17, 19) examined by sampling first 200 / middle 500 / last 200 characters of each Path A bodyTextFull:

  • URL 2 — Anthropic docs (1 843 wc): Path A returned the full “Features overview” page including the Server-side tools table (Advisor tool, Code execution, etc.), the ZDR eligibility matrix, and the footer Terms/Privacy/Support links. Path B (1 460 wc, 0.79 ratio) had the same opening 400 chars; missing words are tail (Help and security, Discord links, “Status” page footer).
  • URL 4 — Wikipedia Knowledge base (1 689 wc): Path A returned the article body including the “Types of Knowledge Base Systems[edit]” section and Versant database mention. Path B (1 552 wc, 0.92 ratio) — near-identical.
  • URL 9 — BBC News (1 891 wc): Path A captured headlines + video metadata (“Watch: Drug dealer jumps fences”, “Drone delivers first Amazon parcels in UK”, etc.) plus “Published 11 Feb” date metadata. Path B (1 355 wc, 0.72 ratio) — opening section captured; lazy-loaded “More from BBC News” sub-trees missing.
  • URL 17 — Forrester research (1 244 wc): Path A captured “Forrester Decisions”, “Forrester Market Insights”, AI-Powered Decision-Making sections. Path B (913 wc, 0.73 ratio) — similar opening; tail “Cookie Settings / LinkedIn / YouTube” boilerplate captured but some mid-page service sub-section text dropped.
  • URL 19 — Cloudflare blog (1 320 wc): Path A captured the recent post stream including “We built our internal AI engineering stack on the same products we ship” (20 million requests, 241 billion tokens). Path B (1 126 wc, 0.85 ratio) — captures same recent posts; missing the deep footer.

Faithfulness verdict on the 5 inspected URLs: Where both paths succeed, Path B’s content is a clean prefix of Path A’s — it captures the head of the rendered DOM and stops short of fully-lazy-loaded sub-trees. For ingestion + chunking + classify purposes the 19 % shortfall is mostly boilerplate; for downstream Q&A / citation purposes it is not load-bearing. However, the spike-plan success criterion is “agent-browser quality ≥ Playwright” — strictly read, Path B’s mean 0.81 ratio is < 1.0 and therefore fails the criterion as stated. A weaker reading (“functional parity for ingest”) would pass; the stricter reading is the one the criterion gate requires.

4. Setup-overhead + maintainability comparison

Section titled “4. Setup-overhead + maintainability comparison”

4.1 Setup overhead

Section titled “4.1 Setup overhead”
StepPath A (Playwright)Path B (agent-browser)
Installbun install (already on KH’s package.json: @playwright/test ^1.58.2)npm install -g agent-browser (global) + agent-browser install (~3.7 GB Chromium download per 0.8.4-pullmd-evaluation.md §2)
Browser provisioningpython3 -m playwright install chromium (per CLAUDE.md gotcha)agent-browser install
Sandbox compatibilityWorks directly (Playwright is a Node-side library)Fails under Claude Code default sandbox — “Socket directory ‘/Users/liamj/.agent-browser’ is not writable: Operation not permitted (os error 1)”. Required dangerouslyDisableSandbox: true for every invocation during this spike.
Per-call configurationOne-off chromium.launch({ headless: true }) + browser.newContext(...)Per-URL --session-name <unique> argument; daemon lifecycle to manage; --session-name <s> close cleanup or sessions accumulate
Output parsingStructured page.evaluate() result is a JSON-safe valueCLI stdout contains ANSI escape codes (e.g. [2m...[0m around the URL) — must be stripped; multi-line body innerText breaks naïve JSON heredocs (this spike’s Path B v1 produced syntactically-invalid JSON requiring a regex-based parser)
Failure modepage.goto() throws on hard errors; networkidle timeout caught with .catch(() => {}) returns whatever renderedCLI exit codes (0 / 124 / etc.); detecting “Cloudflare challenge served instead of real content” requires manual word-count + content-fingerprint heuristics

4.2 Maintainability

Section titled “4.2 Maintainability”
FactorPlaywrightagent-browser
Ecosystem maturityDominant E2E test runner (Microsoft-maintained, 1.58.x); millions of downstream installsv0.13.0 (young); Rust CLI by independent maintainer
KH-team familiarityHigh — KH already runs e2e/tests/** + smoke suite; entire playwright.config.ts infrastructure in treeLow — no prior KH usage; documented in ~/.claude/skills/ but only as ad-hoc browser-automation helper
Documentation depthComprehensive (playwright.dev)--help output (concise but limited); no published reference for wait --load strategies under SPA noise
Stealth / anti-bot postureBattle-tested for E2E; works against Cloudflare on this spike’s sampleTriggers Cloudflare/anti-bot on 4/20 sample URLs; fingerprintable
Existing dependency in KHYES — package.json line 124 "@playwright/test": "^1.58.2"NO — would add new global ops dependency
pullmd integration shapepullmd’s Tier-3 sidecar is already Playwright + FastAPI per 0.8.4-pullmd-evaluation.md §2; swapping it requires forking pullmd, replacing the sidecar with an agent-browser-shaped service, and re-validating pullmd’s X-Source: playwright semanticsLarger surgical change; AGPL-v3 fork risk per 0.8.4-pullmd-evaluation.md §1 confidence-drag

4.3 Net assessment

Section titled “4.3 Net assessment”

Playwright stays cheaper and more reliable than agent-browser on the swap question for three reasons:

  1. KH already pays the Playwright cost for E2E. Removing Playwright is not on offer (E2E suite depends on it); adding agent-browser is therefore strictly additive.
  2. agent-browser’s CLI surface is designed for interactive agent use (vision-model-friendly accessibility snapshots, ref-based selectors, find-by-role) — not for batch URL ingestion. The strengths it has over Playwright (accessibility tree, screenshot annotation, find-by-role) are not utilised by the JS-rendered-fallback ingest role.
  3. Cloudflare bot-detection on 4/20 URLs is a hard quality cliff. Even if every other dimension favoured agent-browser, this single failure mode disqualifies it from the pullmd Tier-3 role where the whole point of Tier-3 is to retrieve content that earlier tiers could not.

5. Decision-gate G11

Section titled “5. Decision-gate G11”

Per 0.9-spike-plan.md §4 (Decision gates rollup row G11):

G11 (NEW): Playwright swap with agent-browser — Adopt swap OR keep Playwright.

Success criteria from §S11:

  • ✗ agent-browser quality ≥ Playwright on sample — FAILED. 14/20 vs 20/20; B/A word-count ratio mean 0.81.
  • ✗ Setup + maintenance overhead lower or equivalent — FAILED. agent-browser adds a new global ops dependency, sandbox-disabled execution, and CLI-output parsing burden; Playwright is already present for E2E.

Decision: KEEP Playwright. OQ6 / CX.7 closed as RESOLVED-NO-SWAP.

6. Downstream-document follow-ups

Section titled “6. Downstream-document follow-ups”

  1. 0.9-decision-graph.md:

    • Row CX.7 (line 167): change status OPEN-PENDING-INVESTIGATION → RESOLVED-NO-SWAP; add evidence anchor S229-S11 spike: 4/20 Cloudflare-bot-block failures; 81% completeness; higher setup overhead; confidence stays 60% → 90% because spike data is now in the document.
    • Row CX.7 (line 297): same status update; add note “Spike S11 (S229) returned NO-SWAP.”
    • §OQ table row OQ6: status flips RE-FRAMED (investigation pending) → RE-FRAMED + RESOLVED-NO-SWAP.

  2. 0.9-intended-architecture.md:

    • §10.1 table line 1124 (agent-browser row): rewrite from “Existing usage; investigate Firecrawl swap” → “Keep agent-browser for ad-hoc browser automation in agent workflows. Playwright stays for pullmd Tier 3 (if pullmd adopted) + KH E2E suite. S229-S11 closed CX.7 as NO-SWAP.”

  3. 0.9-spike-plan.md:

    • §7 risk register row “pullmd quality miss on Cloudflare-protected URLs / Mitigation: Keep agent-browser as Tier 2.5 fallback”: REMOVE this mitigation (the spike disproves it). Replace with “Keep pullmd’s Playwright sidecar; fall back to KH’s existing Firecrawl Tier-3 (lib/intelligence/content-extractor.ts) for Cloudflare-protected URLs where pullmd’s own Cloudflare-native short-circuit doesn’t fire.”
    • §5 outputs table row “0.9-spike-S11-playwright-swap.md”: mark this doc as the completed artefact.
    • §4 decision gates table row G11: mark Adopt swap → KEEP Playwright (S229-S11).

  4. docs/plans/phase-0-investigation/0.8.4-pullmd-evaluation.md:

    • §3.3 capability delta row “pullmd ADDS to KH: Playwright fallback (large win)”: stays — the Playwright fallback is still a real win for KH (KH currently has no Playwright tier in lib/intelligence/content-extractor.ts); the OQ6 question was about whether to further swap pullmd’s Playwright with agent-browser, and the answer is no.

  5. lib/intelligence/content-extractor.ts:

    • No source changes required. The spike confirms KH’s current cascade (RSS → fetch → Jina → Firecrawl → summary_fallback) does not need an agent-browser tier inserted.

7. Residual questions + follow-ups

Section titled “7. Residual questions + follow-ups”

  1. Sample-size sensitivity: 20 URLs is the minimum to detect a 4/20 Cloudflare-block pattern. A 50-URL bake-off (paired with S4 pullmd evaluation) could refine the failure rate. If the prod RSS feed corpus surfaces a different URL distribution (e.g. mostly gov.uk + RSS-feed-embedded content where Cloudflare exposure is < 5 %), the cost-benefit could shift. Action: if S4 pullmd 50-URL bake-off (0.9-spike-S4-pullmd-bakeoff.md) is run later this phase, attach a 30-URL agent-browser comparison column to its results for a combined-corpus view.

  2. agent-browser stealth-mode posture: agent-browser v0.13.0 does not appear to ship a playwright-stealth-equivalent. A future version that did would alter the Cloudflare-block failure rate. Action: re-evaluate G11 if agent-browser ≥ v0.15.0 ships explicit anti-fingerprinting features.

  3. pullmd Tier-3 frequency in KH prod: This spike treated all 20 URLs as if they would route through Tier-3 Playwright. In practice, pullmd’s Cloudflare short-circuit (per 0.8.4-pullmd-evaluation.md §2) handles many of them before Tier-3 fires. Action: S4 should measure actual X-Source: playwright hit-rate on the 50-URL bake-off — if Tier-3 fires on < 10 % of URLs, the cost-impact of the swap question is much smaller than the spike framing implies.

  4. Worktree limitation — no live prod URL sample: This worktree could not access .env.local or live Supabase RSS feed data. A high-fidelity follow-up using actual recent prod URLs from feed_sources + feed_articles would have stronger external validity. Action: if Liam wants to harden the evidence base before Phase 2 commit, re-run this spike from the main worktree (or production-readiness worktree) with 20 URLs drawn from the most-recent feed_articles.url rows where method='firecrawl' (i.e. URLs Tier-2/2.5 already gave up on). Expected outcome unchanged but corpus-validity higher.

  5. Existing-usage of agent-browser in agent workflows: Per CLAUDE.md Gotchas §E2E / Playwright, agent-browser is the recommended tool for “browser testing” in worker-isolated parallel sessions. This spike does not change that recommendation — agent-browser stays the right tool for interactive agent browser-automation tasks. Only the pullmd-Tier-3 swap question closes as NO-SWAP. The two roles do not conflict.

  6. Cost dimension not measured: This spike did not compare $-cost. Playwright is a local-CPU run (no per-call fee); agent-browser is also local. Neither pays a per-call API fee (unlike Firecrawl). So the cost dimension is a tie at zero marginal $-cost; the operational-cost dimension (ops + dev-team time) was assessed above and favours Playwright.

8. Artefacts

Section titled “8. Artefacts”
ArtefactPath
URL corpus (20 entries)/private/tmp/claude/s11/test-urls-v2.json
Path A runner/private/tmp/claude/s11/path-a-playwright.mjs
Path B v1 runner/private/tmp/claude/s11/path-b-agent-browser.sh
Path B v2 runner (no networkidle)/private/tmp/claude/s11/path-b-v2.sh
Bake-off orchestrator/private/tmp/claude/s11/run-bake-off.sh
Per-URL raw results/private/tmp/claude/s11/results/url-{1..20}-{a,b,b2}.json
Aggregate script/private/tmp/claude/s11/aggregate.mjs
Parser (handles agent-browser CLI ANSI + multi-line JSON)/private/tmp/claude/s11/parse-results.mjs

Note on persistence: /private/tmp/ is a tmpfs and will be cleared by macOS at the next reboot. If these artefacts need to be preserved beyond this session, they should be copied into docs/plans/phase-0-investigation/_evidence/0.9-spike-S11/ before session close. The aggregates + per-URL outcomes in §3 are reproducible from the corpus + scripts under /private/tmp/claude/s11/ while they still exist.

9. Confidence + caveats

Section titled “9. Confidence + caveats”

Confidence: 88 % that G11 = KEEP-PLAYWRIGHT is the right call.

Confidence-drag (12 %):

  • (4 %) 20-URL sample is small; a different corpus could surface URL categories where agent-browser’s higher-level abstractions beat raw Playwright. The Cloudflare-bot-block finding is robust at any sample size (it is a categorical capability gap, not a statistical one) but the 81 % completeness ratio could move either direction with a larger sample.
  • (3 %) Worktree could not draw URLs from live KH prod. The substituted corpus is hand-curated to represent the categories pullmd Tier-3 sees; a prod sample might weight categories differently.
  • (3 %) agent-browser v0.13.0 is young. A v0.15 / v0.20 with explicit stealth-mode patches could flip the Cloudflare-bot-block finding. Action: re-evaluate annually or on major agent-browser releases.
  • (2 %) pullmd Tier-3 fire-rate not measured. If pullmd’s Cloudflare-native short-circuit handles 95 %+ of JS-rendered URLs before Tier-3 fires, the swap question is genuinely low-stakes and a future re-evaluation could be deferred indefinitely.

Not confidence-drag (i.e. things this spike does not answer but is not required to):

  • Whether pullmd itself should be adopted (S4 question, not S11).
  • Whether agent-browser is a good tool for agent-driven interactive browser automation (yes — orthogonal role, retained).
  • Whether KH should reduce Firecrawl reliance (separate cost-optimisation question; not gated on S11).

End of S11 spike. G11 = KEEP-Playwright. OQ6 / CX.7 = RESOLVED-NO-SWAP. Phase 0.9 spike phase output 1 of 13.