LLM Reasoning Benchmark
Frontier and open-source models head-to-head on a pinned subset of the public MMLU-Pro reasoning set, re-run on demand as prices and models change. Accuracy is exact-match on the correct option letter; the winner is the cheapest model clearing the accuracy bar (70%). Cost is each model's published list price applied to the actual tokens used, so a cheap-per-token model that reasons at length can still cost more per correct answer.
run 2026-06-22 · run #1 · subset v1 (n=98)| Model | Lab | Accuracy | $ / correct answer | Proj. $/mo @ 1,000,000 | Savings vs incumbent | n scored |
|---|---|---|---|---|---|---|
| Qwen3 235B WINNER | Qwen | 80.6% | $0.0001 | $64.01 | 99% | 98 |
| DeepSeek V4 Flash | DeepSeek | 83.7% | $0.0002 | $201.90 | 95% | 98 |
| Llama 4 Maverick | Meta | 83.7% | $0.0004 | $306.13 | 93% | 98 |
| GPT-5.4 Mini | OpenAI | 77.6% | $0.0008 | $621.83 | 86% | 98 |
| DeepSeek V4 Pro | DeepSeek | 87.8% | $0.0010 | $892.08 | 79% | 98 |
| Grok 4.3 | xAI | 87.8% | $0.0022 | $1,940.27 | 55% | 98 |
| Claude Haiku 4.5 | Anthropic | 77.6% | $0.0025 | $1,958.54 | 55% | 98 |
| GLM 5.2 | Zhipu | 76.5% | $0.0040 | $3,086.64 | 29% | 98 |
| Claude Sonnet 4.6 | Anthropic | 90.8% | $0.0051 | $4,631.69 | -7% | 98 |
| Qwen3.7 Max | Qwen | 91.8% | $0.0065 | $5,942.08 | -35% | 97 (1 errors) |
| GLM 5.1 | Zhipu | 73.2% | $0.0101 | $7,404.82 | -69% | 97 (1 errors) |
| Kimi K2.6 | Moonshot | 75.3% | $0.0099 | $7,426.33 | -69% | 97 (1 errors) |
| Gemini 3.5 Flash | 86.7% | $0.0117 | $10,133.23 | -133% | 98 | |
| GPT-5.5 | OpenAI | 92.9% | $0.0127 | $11,801.58 | -172% | 98 |
| Gemini 3.1 Pro | 85.7% | $0.0165 | $14,171.82 | -226% | 98 | |
| GLM 4.7 Flash | Zhipu | 56.1% | $0.0016 | $891.97 | 79% | 98 |
| Claude Opus 4.8 incumbent | Anthropic | 89.8% | $0.0048 | $4,345.82 | incumbent | 98 |
Run history
| Run | Date | Winner | Winner $/mo | |
|---|---|---|---|---|
| #1 | 2026-06-22 | qwen3-235b | $64.01 | receipts |
Re-run this yourself
Reproduce this exact ranking with the open-source redcrown CLI:
pip install redcrown redcrown build-dataset mmlu-pro --ids-file subset-v1.txt --out exp.json redcrown eval exp.json --concurrency 10
Methodology
Every run executes the same pinned MMLU-Pro subset through each
candidate with the open-source redcrown CLI, routed through a single
OpenRouter gateway so every model runs under identical conditions. Answers are
scored by deterministic exact-match on the option letter, with no LLM judge. Cost
is each model's published native list price (not the gateway markup) applied to the
measured token usage; the monthly projection assumes 1,000,000 questions. Errors and
truncations are counted and shown, never dropped. Caveat: frontier labs may have
trained on MMLU-derived data, so absolute accuracy can be optimistic; the durable
value here is the cost-per-correct ranking and how it shifts as prices and models
change, and the subset can be rotated. Built and run by
RedCrown, which sells no models and routes no
traffic: the proof has no thumb on the scale.