docs/03 · the evidence
Benchmarks & runs
No number, no launch. Every claim below has a receipt: raw per-provider-call records for both harnesses live in docs/testing/data/ as JSON/JSONL, and each report documents its protocol. A number we cannot substantiate is not a number.
Methodology
- Baseline: Browser Use 0.13.6, the open-source default harness, on identical tasks and pages.
- Model: OpenAI gpt-4.1-mini for both harnesses; pricing dated 2026-07-15.
- Tasks: WordPress admin flows at five controlled lengths — WP-N09 through WP-N25 (9 to 25 verified steps).
- 15 complete matched repetitions per harness per cell; a cell only counts when its verify gate passes — 75/75 verified successes per harness. Failed cells are never dropped.
- Intervals: 95% seeded-bootstrap CIs, 10,000 resamples.
- Metric: logical input = uncached input + cache reads + cache writes — so provider caching cannot masquerade as memory reduction.
T10 · Exit gate G1 — the cumulative-token curve PASS
The gate: cumulative tokens must grow materially slower with task length, against a public 30.0% floor. Result: Rote's cumulative logical-input growth is 37.2% slower than Browser Use, 95% CI 35.6–38.8%.
| Cell | Steps | Rote logical input | Browser Use | Reduction [95% CI] |
|---|---|---|---|---|
| WP-N09 | 9 | 47,204 | 55,104 | 14.3% [14.1–14.7] |
| WP-N13 | 13 | 51,068 | 68,455 | 25.4% [25.0–25.7] |
| WP-N17 | 17 | 60,331 | 82,152 | 26.6% [25.7–27.5] |
| WP-N21 | 21 | 69,476 | 95,888 | 27.5% [26.6–28.5] |
| WP-N25 | 25 | 81,203 | 110,131 | 26.3% [25.4–27.2] |
source: docs/testing/T10-g1-cumulative-token-curve.md · data/T10-g1-curve-summary.json
honest caveat
In this frozen pre-cache-key matrix Rote was not cheaper: at WP-N25 its mean billed cost was 5.4% higher ($0.0310 vs $0.0294) and p50 latency 6.4% higher (72.0s vs 67.7s), because Browser Use received more discounted cache reads. That gap is what T11 was run to close.T11 · Cache-key economics — the cost win
One change: route a SHA-256 of the immutable prefix through OpenAI's prompt_cache_key — no added prompt text. At WP-N25 that cuts Rote's mean bill 20.5% and makes it 16.0% cheaper than Browser Use, while preserving ~37.6% slower logical growth.
| Cell | Cost before | After | Cut [95% CI] | vs Browser Use | Cache reads |
|---|---|---|---|---|---|
| WP-N09 | $0.0193 | $0.0168 | 12.6% [1.7–24.6] | −15.0% [−29.4–1.3] | 981 → 8,704 |
| WP-N13 | $0.0193 | $0.0121 | 37.2% [22.2–49.7] | 34.4% [20.2–46.5] | 6,110 → 30,276 |
| WP-N17 | $0.0235 | $0.0201 | 14.7% [2.3–27.2] | 8.8% [−2.6–21.7] | 5,649 → 16,503 |
| WP-N21 | $0.0270 | $0.0236 | 12.6% [5.0–21.6] | 8.3% [0.3–17.5] | 6,707 → 18,790 |
| WP-N25 | $0.0310 | $0.0246 | 20.5% [11.3–30.3] | 16.0% [6.2–26.2] | 10,377 → 29,722 |
source: docs/testing/T11-cache-key-economics.md · data/T11-cache-key-economics-summary.json
WP-N09, the shortest cell, still loses on cost and its interval crosses parity — this is a long-task cache win, not a universal one.
Supporting measurements
A4 · diff observations
Across the G1 matrix, A4 emitted 849 diffs (median 24 chars) against 240 grounded bootstraps (median 9,270 chars) — a median render-size reduction of 99.6% relative to re-sending the page.
source: docs/testing/T10-g1-cumulative-token-curve.md
T2 · observation stability
Real-page scale: WordPress /wp-admin/edit.php with 120 seeded posts and 100 rows per page — 6,784 captured elements distill to 797 nodes (787 actionable), 89,114 rendered chars ≈ 22,279 tokens, with zero measured variance across 15 repetitions.
source: docs/testing/data/T2-wordpress-observation-stability.json
T3 · OpenAI cache preflight
86 measurement calls before the layout work: prompt tokens min 539 / median 806 / max 21,433; only 2 of 26 cache-eligible calls hit (7.7%). Decision: go do the layout work — which became B3 and T11.
source: docs/testing/data/T3-openai-cache-preflight.json
T1 · the false-negative lesson
Fixture B2 went 0/7 to 11/11 on two models after making per-step expect optional — the harness was failing verified-correct runs on over-strict assertions. The bug became a test in the same PR (issues #49/#50).
source: docs/testing/02-architecture.md §Status
The two exit gates
G1 · the curve
pass · 37.2%
Cumulative tokens grow materially slower with task length, ≥30% floor, CI-bounded.
G2 · the level
not yet run
A tokens-per-task win at success parity, ≥15 runs, bootstrap lower bound above the floor. The launch package waits on it.