Skip to main content

Benchmark · 6 tasks · 4 runs each

Structured specs cut page tokens by two thirds

The same six page requests, written once as raw HTML and once as a Page Spec, measured on tokens, latency and rubric accuracy.

Headline comparison

Raw HTML against Page Spec

Page Spec vs Raw HTML4 improved, 1 unchanged

Raw HTML against Page Spec
MetricRaw HTMLPage SpecChangeVerdict
Output tokens per pagelower is better17,3005,517−11,783−68.1%improved
Generation timelower is better41.2 s14.8 s−26.4 s−64.1%improved
Rubric accuracyhigher is better91.0%97.5%+6.5%+7.1%improved
Cost per pagelower is better$0.26$0.08−$0.18−68.1%improved
Retries to a valid pagelower is better1.41.40.00%unchanged
Method
Each task ran four times per arm on the same model and temperature. Tokens are output tokens only; the spec arm includes the compact describe output it read.
Limitations
Six tasks is a small sample, and accuracy is a rubric score from one reviewer.

Evidence

Per-task results

Tokens

Output tokens per task
Output tokens per taskRaw HTML: Dashboard 21.8k, Incident 18.4k, Release 9.6k, Architecture 26.2k. Page Spec: Dashboard 6.4k, Incident 5.9k, Release 3.1k, Architecture 8.8k010k20k30kDashboardIncidentReleaseArchitectureDashboard: Raw HTML 21.8kIncident: Raw HTML 18.4kRelease: Raw HTML 9.6kArchitecture: Raw HTML 26.2kDashboard: Page Spec 6.4kIncident: Page Spec 5.9kRelease: Page Spec 3.1kArchitecture: Page Spec 8.8kTokensTask

Raw HTML: Dashboard 21.8k, Incident 18.4k, Release 9.6k, Architecture 26.2k. Page Spec: Dashboard 6.4k, Incident 5.9k, Release 3.1k, Architecture 8.8k

Chart data table
Output tokens per task
TaskarmTokens
DashboardRaw HTML21.8k
DashboardPage Spec6.4k
IncidentRaw HTML18.4k
IncidentPage Spec5.9k
ReleaseRaw HTML9.6k
ReleasePage Spec3.1k
ArchitectureRaw HTML26.2k
ArchitecturePage Spec8.8k

Table

Tokens per task

Tokens per task
TaskRaw HTMLPage SpecSavingAccuracySpec tokens by run
Dashboard21,8006,40071%98%7,100, 6,800, 6,500, 6,400
Incident report18,4005,90068%97%6,600, 6,200, 6,000, 5,900
Release notes9,6003,10068%100%3,500, 3,300, 3,200, 3,100
Architecture review26,2008,80066%95%9,900, 9,400, 9,000, 8,800
Pricing page15,1004,30072%99%4,900, 4,600, 4,400, 4,300
Research summary12,7004,60064%96%5,200, 4,900, 4,700, 4,600

Cost bridge

Monthly cost bridge for 2,000 pages
Monthly cost bridge for 2,000 pagesCost: Raw HTML +$520.00, Fewer tokens -$372.00, Fewer retries -$12.00, Describe reads +$18.00, Validation +$12.00; total $166.00.$0$200$400$600Raw HTMLFewer tokensFewer retriesDescribe readsValidationTotalRaw HTML: +$520.00, running total $520.00Fewer tokens: -$372.00, running total $148.00Fewer retries: -$12.00, running total $136.00Describe reads: +$18.00, running total $154.00Validation: +$12.00, running total $166.00Total: $166.00Cost

Cost: Raw HTML +$520.00, Fewer tokens -$372.00, Fewer retries -$12.00, Describe reads +$18.00, Validation +$12.00; total $166.00.

Chart data table
Monthly cost bridge for 2,000 pages
StepChangeRunning total
Raw HTML+$520.00$520.00
Fewer tokens-$372.00$148.00
Fewer retries-$12.00$136.00
Describe reads+$18.00$154.00
Validation+$12.00$166.00
Total$166.00

Describe reads and validation add a small fixed cost per page; the token saving dominates.

Sources

  1. Benchmark run log, four runs per task and arm

    Internal evaluation notebook · 2026

    Raw token counts, timings and retry counts per run.

  2. Page quality rubric v2

    Evaluation guild · 2026

    Ten checks covering structure, accessibility and content fidelity.

  3. Model list prices, September 2026

    Provider pricing page · 2026