Kagi Reasoning Benchmark
Kagi's private, regularly rotated set of tricky reasoning, coding, math and instruction-following questions, many translated into other languages, built to resist memorization.
| Category | Reasoning |
|---|---|
| Metric | Accuracy (%) |
| Source | Kagi (original) |
| License | None stated |
| Benchmark page | help.kagi.com |
| Status | Scored |
| Weight | 0.8 within Reasoning |
| Fit | Discrimination 0.74, noise 0.37 |
Results
| # | Model | Provider | Result | Date |
|---|---|---|---|---|
| 1 | Claude Fable 5 Thinking | Anthropic | 91.4% | 2026-11-11 |
| 2 | Claude Fable 5 | Anthropic | 88.8% | 2026-11-11 |
| 3 | GPT-5.5 | OpenAI | 88.8% | 2026-11-11 |
| 4 | Claude Opus 4.8 Thinking | Anthropic | 88.8% | 2026-11-11 |
| 5 | Claude Opus 4.6 Thinking | Anthropic | 83.6% | 2026-11-11 |
| 6 | Grok 4.5 | xAI | 83.5% | 2026-11-11 |
| 7 | Claude Opus 4.7 Thinking | Anthropic | 80.7% | 2026-11-11 |
| 8 | Claude Opus 4.5 (latest) Thinking | Anthropic | 80.2% | 2026-11-11 |
| 9 | Gemini 3 Pro Preview | 80.1% | 2026-11-11 | |
| 10 | Kimi K2.5 Thinking | Moonshot AI | 78.5% | 2026-11-11 |
| 11 | GPT-5 Pro | OpenAI | 76.8% | 2026-11-11 |
| 12 | Grok 4.20 (Reasoning) | xAI | 75.0% | 2026-11-11 |
| 13 | GLM-5 Thinking | Zhipu AI | 75.0% | 2026-11-11 |
| 14 | Claude Opus 4 Thinking | Anthropic | 74.3% | 2026-11-11 |
| 15 | Grok 4 | xAI | 73.6% | 2026-11-11 |
| 16 | Claude Opus 4.7 | Anthropic | 73.3% | 2026-11-11 |
| 17 | GPT-5.2 | OpenAI | 73.3% | 2026-11-11 |
| 18 | Claude Sonnet 4 Thinking | Anthropic | 73.0% | 2026-11-11 |
| 19 | GPT-5 | OpenAI | 72.7% | 2026-11-11 |
| 20 | Qwen3 Max | Alibaba | 72.5% | 2026-11-11 |
| 21 | Claude Opus 4.6 | Anthropic | 72.4% | 2026-11-11 |
| 22 | o3-pro | OpenAI | 72.1% | 2026-11-11 |
| 23 | Claude Opus 4.5 | Anthropic | 70.7% | 2026-11-11 |
| 24 | GPT-5 Mini | OpenAI | 70.3% | 2026-11-11 |
| 25 | GPT-5-Codex | OpenAI | 70.3% | 2026-11-11 |
| 26 | Gemini 2.5 Pro | 70.3% | 2026-11-11 | |
| 27 | DeepSeek-R1 | DeepSeek | 69.4% | 2026-11-11 |
| 28 | o4-mini | OpenAI | 67.6% | 2026-11-11 |
| 29 | o3 | OpenAI | 67.6% | 2026-11-11 |
| 30 | Gemini 3.1 Flash Lite | 67.2% | 2026-11-11 | |
| 31 | GPT-5.6 Sol | OpenAI | 67.0% | 2026-11-11 |
| 32 | Qwen3-Next 80B-A3B (Thinking) | Alibaba | 66.7% | 2026-11-11 |
| 33 | Grok 4 Fast Thinking | xAI | 66.1% | 2026-11-11 |
| 34 | Kimi K2 Thinking | Moonshot AI | 64.4% | 2026-11-11 |
| 35 | Kimi K2.5 | Moonshot AI | 63.8% | 2026-11-11 |
| 36 | GPT-5.4 | OpenAI | 63.8% | 2026-11-11 |
| 37 | Gemma 4 31B IT | 63.5% | 2026-11-11 | |
| 38 | Qwen Plus | Alibaba | 63.3% | 2026-11-11 |
| 39 | GLM-5.2 | Zhipu AI | 62.6% | 2026-11-11 |
| 40 | GPT-5 Nano | OpenAI | 62.2% | 2026-11-11 |
| 41 | Grok 3 | xAI | 61.3% | 2026-11-11 |
| 42 | Grok-3 mini | xAI | 61.3% | 2026-11-11 |
| 43 | GLM-5.2 Thinking | Zhipu AI | 60.0% | 2026-11-11 |
| 44 | GLM-4.5V | Zhipu AI | 59.8% | 2026-11-11 |
| 45 | Claude Opus 4 | Anthropic | 59.6% | 2026-11-11 |
| 46 | GPT OSS 120B | OpenAI | 58.6% | 2026-11-11 |
| 47 | Claude Sonnet 4.5 | Anthropic | 57.9% | 2026-11-11 |
| 48 | GLM-4.5 | Zhipu AI | 57.9% | 2026-11-11 |
| 49 | MiniMax-M2 | MiniMax | 57.8% | 2026-11-11 |
| 50 | DeepSeek-V3.1-Terminus | DeepSeek | 57.4% | 2026-11-11 |
| 51 | Gemini 2.5 Flash Thinking | 56.8% | 2026-11-11 | |
| 52 | Claude Sonnet 4 | Anthropic | 55.9% | 2026-11-11 |
| 53 | Llama 4 Maverick | Meta | 55.9% | 2026-11-11 |
| 54 | Hermes 4 405B Thinking | Nousresearch | 55.8% | 2026-11-11 |
| 55 | MiniMax-M2.5 | MiniMax | 55.2% | 2026-11-11 |
| 56 | Qwen3-30B-A3B-Thinking (Jul 2025) | Alibaba | 54.9% | 2026-11-11 |
| 57 | Qwen3-Next 80B-A3B Instruct | Alibaba | 54.4% | 2026-11-11 |
| 58 | DeepSeek V4 Pro | DeepSeek | 53.5% | 2026-11-11 |
| 59 | GPT OSS 20B | OpenAI | 53.2% | 2026-11-11 |
| 60 | DeepSeek V3.1 | DeepSeek | 53.2% | 2026-11-11 |
| 61 | x-AI/Grok-Code-Fast 1 | xAI | 52.5% | 2026-11-11 |
| 62 | GPT-4.1 | OpenAI | 52.3% | 2026-11-11 |
| 63 | DeepSeek-R1-Distill-Llama-70B | DeepSeek | 52.3% | 2026-11-11 |
| 64 | DeepSeek V4 Flash | DeepSeek | 52.2% | 2026-11-11 |
| 65 | DeepSeek-V3.2-Exp | DeepSeek | 52.2% | 2026-11-11 |
| 66 | Nemotron Nano 12B v2 VL | NVIDIA | 51.7% | 2026-11-11 |
| 67 | GLM-5 | Zhipu AI | 51.7% | 2026-11-11 |
| 68 | GPT-5.6 Terra | OpenAI | 51.3% | 2026-11-11 |
| 69 | Mistral Large 3 | Mistral | 50.9% | 2026-11-11 |
| 70 | Mistral Medium | Mistral | 50.0% | 2026-11-11 |
| 71 | GPT-5.6 Luna | OpenAI | 49.1% | 2026-11-11 |
| 72 | Qwen3-14B | Alibaba | 49.1% | 2026-11-11 |
| 73 | Qwen3 32B Non-reasoning | Alibaba | 48.7% | 2026-11-11 |
| 74 | GPT-4.1 mini | OpenAI | 48.6% | 2026-11-11 |
| 75 | Nemotron Nano 9B v2 | NVIDIA | 47.4% | 2026-11-11 |
| 76 | GLM-4.6 | Zhipu AI | 45.7% | 2026-11-11 |
| 77 | ERNIE 4.5 300B A47B | Baidu | 45.0% | 2026-11-11 |
| 78 | Gemini 2.5 Flash | 44.1% | 2026-11-11 | |
| 79 | GLM-4.5-Air | Zhipu AI | 43.0% | 2026-11-11 |
| 80 | Mistral Medium 3.5 | Mistral | 41.4% | 2026-11-11 |
| 81 | Mistral Small 4 | Mistral | 40.5% | 2026-11-11 |
| 82 | Gemini 2.5 Flash-Lite | 40.5% | 2026-11-11 | |
| 83 | Mistral Small 3.2 24B | Mistral | 40.4% | 2026-11-11 |
| 84 | GPT-5.4 nano | OpenAI | 39.7% | 2026-11-11 |
| 85 | GPT-5.4 mini | OpenAI | 37.9% | 2026-11-11 |
| 86 | Mistral Small (latest) | Mistral | 37.8% | 2026-11-11 |
| 87 | Phi-4-Reasoning-plus | Microsoft | 37.8% | 2026-11-11 |
| 88 | Devstral Small | Mistral | 37.7% | 2026-11-11 |
| 89 | Devstral Medium | Mistral | 37.7% | 2026-11-11 |
| 90 | Llama 4 Scout | Meta | 36.9% | 2026-11-11 |
| 91 | Qwen2.5-VL 72B Instruct | Alibaba | 36.0% | 2026-11-11 |
| 92 | Grok 4 Fast | xAI | 35.4% | 2026-11-11 |
| 93 | Gemma 3 27B IT | 35.1% | 2026-11-11 | |
| 94 | Llama 3-70B | Meta | 35.1% | 2026-11-11 |
| 95 | Claude Haiku 3 | Anthropic | 34.2% | 2026-11-11 |
| 96 | GPT-4.1 nano | OpenAI | 33.3% | 2026-11-11 |
| 97 | Codestral 25.08 | Mistral | 32.5% | 2026-11-11 |
| 98 | Gemma 3n E4B IT | 31.5% | 2026-11-11 | |
| 99 | Hermes 4 405B Non-reasoning | Nousresearch | 30.1% | 2026-11-11 |
| 100 | GPT-4o mini | OpenAI | 28.8% | 2026-11-11 |
| 101 | Command A | Cohere | 28.8% | 2026-11-11 |
| 102 | Gemma 3 4B IT | 25.2% | 2026-11-11 | |
| 103 | Phi-4-multimodal-instruct | Microsoft | 17.1% | 2026-11-11 |
| 104 | Magistral Small | Mistral | 6.3% | 2026-11-11 |