Skip to main content

Data tables

Benchmark runs

One dataset, typed columns, sorting on every header and search past ten rows.

All runs by p95 latency

24 rows

All runs by p95 latency
ModelProviderTokensSavingCost / 1Mp95StatusCoverageReleasedSix-week trend
atlas-largeNorthwind1.8k35%$0.12180 mspass48%Jan 1, 202630, 47, 64, 81, 38, 55
helix-deepHelix120.6k50%$4.07235 mspass67%Apr 26, 202645, 62, 79, 36, 53, 70
atlas-miniNorthwind9.7k72%$0.65277 mspass77%Jun 12, 202643, 60, 77, 34, 51, 68
iris-visionIris128.5k87%$0.60332 mspass96%Sep 9, 202658, 75, 32, 49, 66, 83
beacon-7bHarbor Labs17.6k49%$1.18374 msfail54%Feb 23, 202656, 73, 30, 47, 64, 81
iris-textIris136.4k64%$1.13429 msflaky73%May 20, 202671, 88, 45, 62, 79, 36
beacon-13bHarbor Labs25.6k86%$1.71471 mspass83%Jul 6, 202669, 86, 43, 60, 77, 34
juniper-1Juniper144.3k41%$1.66526 mspass50%Jan 3, 202684, 41, 58, 75, 32, 49
cedar-proCedar AI33.5k63%$2.24568 msflaky60%Mar 17, 202682, 39, 56, 73, 30, 47
juniper-2Juniper152.3k78%$2.19623 mspass79%Jun 14, 202637, 54, 71, 88, 45, 62
cedar-liteCedar AI41.4k40%$2.77665 mspass89%Aug 28, 202635, 52, 69, 86, 43, 60
kestrel-edgeKestrel160.2k55%$2.72720 mspass56%Feb 25, 202650, 67, 84, 41, 58, 75
delta-coderDelta49.3k77%$3.30762 mspass66%Apr 11, 202648, 65, 82, 39, 56, 73
kestrel-cloudKestrel168.1k92%$3.25817 msfail85%Jul 8, 202663, 80, 37, 54, 71, 88
delta-chatDelta57.2k54%$3.83859 msfail95%Sep 22, 202661, 78, 35, 52, 69, 86
lumen-rLumen176k69%$3.78914 mspass62%Mar 19, 202676, 33, 50, 67, 84, 41
ember-xlEmber65.2k91%$0.36956 mspass72%May 5, 202674, 31, 48, 65, 82, 39
lumen-qLumen183.9k46%$0.311 spass91%Aug 2, 202689, 46, 63, 80, 37, 54
ember-sEmber73.1k68%$0.891.1 spass49%Jan 16, 202687, 44, 61, 78, 35, 52
fjord-baseFjord81k45%$1.421.1 sflaky78%Jun 27, 202640, 57, 74, 31, 48, 65
fjord-instructFjord88.9k82%$1.951.2 spass55%Feb 10, 202653, 70, 87, 44, 61, 78
granite-v2Quarry96.8k59%$2.481.3 spass84%Jul 21, 202666, 83, 40, 57, 74, 31
granite-v3Quarry104.7k36%$3.011.4 spass61%Mar 4, 202679, 36, 53, 70, 87, 44
helix-fastHelix112.7k73%$3.541.5 sfail90%Aug 15, 202632, 49, 66, 83, 40, 57

Top five savings

The five models with the largest token savings.
ModelSavingStatus
kestrel-cloud92%fail
ember-xl91%pass
iris-vision87%pass
beacon-13b86%pass
fjord-instruct82%pass

Release checklist

Release checklist
stepownerhours
Freeze the datasetdata3
Re-run the suiteeval6.5
Publish the reportdocs1.25