Four decision models, 60 judgment calls
Four AI decision models answered the same 60 judgment calls, one request per question, against two simple rule baselines on 22 Oct 2026.
| Approach | Correct (of 60) | Accuracy | Accuracy |
|---|---|---|---|
| Decision models, one request per question | |||
| SubQsubq-snap-preview | 53 | 88% | |
| JEVTypeSafe AI · jev-1.13.0 | 52 | 87% | |
| mammoth-1Baseten · Mammoth 27B | 49 | 82% | |
| elephant-1Baseten · Elephant 9B | 48 | 80% | |
| Simple rule baselines | |||
| Keyword rule (baseline)Picks the answer by matching keywords | 37 | 62% | |
| Majority answer (baseline)Always gives the most common answer | 25 | 42% | |
What we asked
By format: 30 yes/no, 12 multiple choice, 18 scored on a scale.
Where the models differed
All four models got 29 of 30 yes/no questions right. JEV also got one more multiple-choice answer right, and most of the spread comes from the 18 scored questions (8 to 13 right).
| Approach | Yes/no (of 30) | Multiple choice (of 12) | Scored (of 18) | Total (of 60) | Missed |
|---|---|---|---|---|---|
| SubQ | 29 | 11 | 13 | 53 | 7 |
| JEV | 29 | 12 | 11 | 52 | 8 |
| mammoth-1 | 29 | 11 | 9 | 49 | 11 |
| elephant-1 | 29 | 11 | 8 | 48 | 12 |
| Keyword rule | 23 | 8 | 6 | 37 | 23 |
| Majority answer | 15 | 4 | 6 | 25 | 35 |
Is the top of the table a real difference?
No. Comparing each model with JEV question by question (two-sided exact McNemar test), none of the gaps is significant:
- SubQ p = 1.00 (2 questions only SubQ got right, 1 only JEV)
- mammoth-1 p = 0.25 (0 vs 3)
- elephant-1 p = 0.12 (0 vs 4)
With 60 questions, one or two answers can reorder the top.
Tell us what your team does by hand, or what you want it to do. We'll reply with how we would build it.