Artificial Analysis Finally Scored Calibration. OpenAI's Decisions API Lands 13th of 17 Decision Models, Below Jev, While the Same Luna Run as a Plain LLM Scores 76.
Artificial Analysis released Decision Index v2 on November 8 with a metric that scores correctness and confidence together, 50 times one plus accuracy minus Brier, across six evaluations, 45% of them private. GPT-6 Luna through OpenAI's Decisions API scores 61.7, thirteenth of seventeen decision models and behind TypeSafe's Jev 1.14 at 67.0; the same GPT-6 Luna run as an ordinary LLM at max effort scores 76.4, second overall behind Claude Haiku 5.5 at 77.1. The top decision models, Drex Max at 73.4, GLiDE at 72.8, and Mammoth 27B at 72.4, cost a fraction of the LLMs per task and run in milliseconds; one of the six evaluations is built by TypeSafe, which sells Jev. v1 and v2 scores are not comparable. We read AA's article and evaluation page.
By T.W. Ghost
Three times this fall we wrote that OpenAI publishes no calibration data for its Decisions API, the typed-answer endpoint that returns a probability with every choice. Today, November 8, 2026, Artificial Analysis published a board that measures it. Decision Index v2 scores each answer on correctness and confidence together: 50 times one plus the correctness minus the squared gap between the stated probability and the outcome, which works out to 50 times one plus accuracy minus Brier. A model that is right and sure scores high; a model that is right and hedging, or wrong and sure, scores lower.
The result for OpenAI's product is the catch. GPT-6 Luna through the Decisions API scores 61.7, thirteenth of seventeen dedicated decision models, below TypeSafe's Jev 1.14 at 67.0 and below Microsoft-Decision-1 at 62.8. The same GPT-6 Luna, run as an ordinary language model at max effort, scores 76.4, second on the whole board behind Claude Haiku 5.5 at 77.1. The model is not the problem. The packaging is, or the way the Decisions API reports its probabilities. The other catch is money and time: the LLMs that top the board cost fifteen to sixty-one times more per task than the best dedicated decision model and run forty to one hundred twenty times slower. We read AA's article and its evaluation page. Here is what held up.
The Board, by Who and What
| Model | Decision Index v2 | Cost per 1M tasks | Time per task |
|---|---|---|---|
| Claude Haiku 5.5 (max), LLM | 77.1 | $6,240 | 32.65 s |
| GPT-6 Luna (max), LLM | 76.4 | $1,530 | 11.34 s |
| Drex Max v1.1 (Nace AI) | 73.4 | $102 | 275 ms |
| GLiDE v1.2 (Fastino) | 72.8 | $780 | |
| Mammoth 27B (Base Labs) | 72.4 | ||
| GPT-6 Luna (low), LLM | 69.2 | ||
| Jev 1.14 (TypeSafe) | 67.0 | 92 ms | |
| Claude Haiku 5.5 (low), LLM | 66.7 | ||
| Microsoft-Decision-1 | 62.8 | ||
| GPT-6 Luna via Decisions API | 61.7, 13th of 17 decision models | ||
| d1; Decider Flash; Clef; Clef-Omni | 61.6; 61.6; 60.2; 59.3 | Decider v1.2 $68; Clef $859 | Celeris-1.1 $14 at 56 ms; Elephant 9B 70 ms |
The product sits near the bottom and its own model sits near the top. Luna as an LLM at 76.4 and Luna through the Decisions API at 61.7 are on the same board, measured the same way. That gap is the number OpenAI should be asked about.
Calibration costs money in both directions. Haiku 5.5 at max is the most calibrated thing on the board and costs $6,240 per million tasks; Drex Max scores 73.4 at $102 and 275 milliseconds. For a routing layer that is the whole trade.
Billing shape beats list price. GLiDE bills the document input on every question, so it costs about eight times Drex Max per task despite a list price only about a third higher. AA's own note, and the kind of thing a per-token comparison hides.
What Is in the Index
| Evaluation | Weight | Note |
|---|---|---|
| AA-Paperclip | 30% | 1,500 private documents; a document counts only if all 15 answers are right; pass rates Haiku 5.5 max 92%, Luna max 86%, Drex Max 63%, GLiDE 62%, Decider 55% |
| EvalSafe-O*NET-AA | 15% | Built by TypeSafe, which sells Jev |
| When2Call | 10% | |
| AA-Omniscience-Decision v1.1 | 15% | |
| HLE-MultiChoice v1.1 | 15% | |
| AA-Symmetry | 15% | Rewards abstention; AA notes Base Labs models lead partly by confidently saying they do not know |
| WildGuardTest | removed | |
| Private share | 45% | Up from 25% in v1 |
Three honest observations:
- v1 and v2 do not compare. Luna through the Decisions API scored 48.5 and eleventh of thirteen on v1 in October, and Jev 57.1. New metric, new weights, more private data; "Luna rose from 48.5 to 61.7" is a sentence that should not be written.
- One benchmark belongs to a vendor on the board. TypeSafe built EvalSafe-O*NET-AA and sells Jev. AA discloses it; so do we. It is 15% of the score.
- Forty-five percent private is the same trade as Intelligence Index v5. Harder to contaminate, impossible to reproduce. Readers get AA's judgment on data they cannot see, which is the same caveat we gave the main index.
Verified vs Unconfirmed: The Scorecard
| Claim | Verdict |
|---|---|
| Decision Index v2 released Nov 8, 2026; metric definition; weights; 45% private | Verified (AA article; evaluation page) |
| Scores for decision models and LLMs as listed | Verified (AA evaluation page) |
| Luna via Decisions API 13th of 17 decision models | Our count from AA's page data |
| Cost per 1M tasks and time per task figures | Verified (AA article) |
| GLiDE per-document billing makes it about 8x Drex Max per task | Verified (AA's note) |
| Paperclip pass rates | Verified (AA article) |
| v1 figures: Luna-Decisions 48.5, Jev 57.1 | Our October log; not comparable to v2 |
| EvalSafe-O*NET-AA built by TypeSafe | Verified (AA evaluation page) |
Who Should Use It
Use a dedicated decision model if your routing layer needs calibrated probabilities at scale: Drex Max scores 73.4 against the best LLM's 77.1, at $102 per million tasks against $6,240 and in 275 milliseconds against 32 seconds, by AA's measurement.
Skip the Decisions API for calibration until OpenAI explains why its packaging scores so far below its own model, and skip any comparison that mixes v1 and v2 numbers.
The frontier is elsewhere: the most calibrated model on the board is a general LLM at max effort, and the second most calibrated is the one OpenAI wraps in the product that lands thirteenth. Our model comparison has the current standings side by side.
Update, November 12, 2026
- Google joined the board, and the board moved to v2.1. On November 11 Google launched Gemini Decision 1, and AA scored it the same day on Decision Index v2.1: 66, level with Jev 1.14 and Elephant 9B, seven points behind Drex Max v1.1 and GLiDE v1.2 at 73, at $345 per million tasks against $103 for Drex Max and $339 for GPT-6 Luna through OpenAI's Decisions API. The v2.1 label means the table above, which is v2, should not be read next to those rows without checking for rescoring; Drex Max's cost line read $102 here and $103 in AA's new post.
Update, November 13, 2026
- Nov 12: the cheap row near the top is now open weights. Perplexity's Decider v1.3, Apache 2.0 on a Qwen3.8-27B base, scores 70 on v2.1, sixth of 21, at $68 per million tasks and 289 milliseconds; Decider Flash v1.3 scores 64 at $34 and 178 milliseconds. Decider v1.2 also reads 70 at $68, seventh of 21, in 229 milliseconds, so the new version is flat on this board and slower. On the same v2.1 pages Gemini Decision 1 is eleventh of 21 at 66, 211 milliseconds, and $345, and the board median is 66, 206 milliseconds, and $137 per million tasks. Drex Max v1.1 and GLiDE v1.2 still lead at 73.
Sources
- Artificial Analysis: Decision Index v2
- Artificial Analysis: Decision Index evaluation page
- Artificial Analysis on X: Decision Index v2
- Artificial Analysis on X: Gemini Decision 1 on Decision Index v2.1
- Artificial Analysis: pplx-decider-v1.3
*Not sure whether your branching logic needs a decision model or a calibrated LLM? Take the free 2-minute quiz and get matched. Then read the beta post where we first asked for the calibration number: OpenAI's Decisions API.*