Benchmark · 6 tasks · 4 runs each
Structured specs cut page tokens by two thirds
The same six page requests, written once as raw HTML and once as a Page Spec, measured on tokens, latency and rubric accuracy.
Headline comparison
Raw HTML against Page Spec
Page Spec vs Raw HTML4 improved, 1 unchanged
| Metric | Raw HTML | Page Spec | Change | Verdict |
|---|---|---|---|---|
| Output tokens per pagelower is better | 17,300 | 5,517 | −11,783−68.1% | improved |
| Generation timelower is better | 41.2 s | 14.8 s | −26.4 s−64.1% | improved |
| Rubric accuracyhigher is better | 91.0% | 97.5% | +6.5%+7.1% | improved |
| Cost per pagelower is better | $0.26 | $0.08 | −$0.18−68.1% | improved |
| Retries to a valid pagelower is better | 1.4 | 1.4 | 0.00% | unchanged |
- Method
- Each task ran four times per arm on the same model and temperature. Tokens are output tokens only; the spec arm includes the compact describe output it read.
- Limitations
- Six tasks is a small sample, and accuracy is a rubric score from one reviewer.
- Source
- Run log and rubric
Evidence
Per-task results
Tokens
Raw HTML: Dashboard 21.8k, Incident 18.4k, Release 9.6k, Architecture 26.2k. Page Spec: Dashboard 6.4k, Incident 5.9k, Release 3.1k, Architecture 8.8k
Chart data table
| Task | arm | Tokens |
|---|---|---|
| Dashboard | Raw HTML | 21.8k |
| Dashboard | Page Spec | 6.4k |
| Incident | Raw HTML | 18.4k |
| Incident | Page Spec | 5.9k |
| Release | Raw HTML | 9.6k |
| Release | Page Spec | 3.1k |
| Architecture | Raw HTML | 26.2k |
| Architecture | Page Spec | 8.8k |
Table
Tokens per task
| Task | Raw HTML | Page Spec | Saving | Accuracy | Spec tokens by run |
|---|---|---|---|---|---|
| Dashboard | 21,800 | 6,400 | 71% | 98% | 7,100, 6,800, 6,500, 6,400 |
| Incident report | 18,400 | 5,900 | 68% | 97% | 6,600, 6,200, 6,000, 5,900 |
| Release notes | 9,600 | 3,100 | 68% | 100% | 3,500, 3,300, 3,200, 3,100 |
| Architecture review | 26,200 | 8,800 | 66% | 95% | 9,900, 9,400, 9,000, 8,800 |
| Pricing page | 15,100 | 4,300 | 72% | 99% | 4,900, 4,600, 4,400, 4,300 |
| Research summary | 12,700 | 4,600 | 64% | 96% | 5,200, 4,900, 4,700, 4,600 |
Cost bridge
Cost: Raw HTML +$520.00, Fewer tokens -$372.00, Fewer retries -$12.00, Describe reads +$18.00, Validation +$12.00; total $166.00.
Chart data table
| Step | Change | Running total |
|---|---|---|
| Raw HTML | +$520.00 | $520.00 |
| Fewer tokens | -$372.00 | $148.00 |
| Fewer retries | -$12.00 | $136.00 |
| Describe reads | +$18.00 | $154.00 |
| Validation | +$12.00 | $166.00 |
| Total | $166.00 |
Describe reads and validation add a small fixed cost per page; the token saving dominates.
Sources
Benchmark run log, four runs per task and arm
Raw token counts, timings and retry counts per run.
Page quality rubric v2
Ten checks covering structure, accessibility and content fidelity.
Model list prices, September 2026