Aider Polyglot
225 hard Exercism exercises in six languages, solved in the Aider harness. No longer updated, so newer models are missing.
| Category | Coding |
|---|---|
| Metric | Percent correct |
| Source | Epoch AI (evaluator) |
| License | CC BY 4.0 |
| Benchmark page | aider.chat |
| Status | Scored |
| Weight | 0.7 within Coding |
| Fit | Discrimination 0.90, noise 0.12 |
Results
| # | Model | Provider | Result | Date |
|---|---|---|---|---|
| 1 | GPT-5 High | OpenAI | 88.0% | 2026-11-11 |
| 2 | GPT-5 Medium | OpenAI | 86.7% | 2026-11-11 |
| 3 | o3-pro High | OpenAI | 84.9% | 2026-11-11 |
| 4 | Gemini 2.5 Pro Preview 06-05 Thinking | 83.1% | 2026-11-11 | |
| 5 | o3 High | OpenAI | 81.3% | 2026-11-11 |
| 6 | GPT-5 Low | OpenAI | 81.3% | 2026-11-11 |
| 7 | Grok 4 | xAI | 79.6% | 2026-11-11 |
| 8 | Grok 4 High | xAI | 79.6% | 2026-11-11 |
| 9 | Gemini 2.5 Pro Preview 06-05 | 79.1% | 2026-11-11 | |
| 10 | o3 | OpenAI | 76.9% | 2026-11-11 |
| 11 | o3 Medium | OpenAI | 76.9% | 2026-11-11 |
| 12 | DeepSeek Reasoner | DeepSeek | 74.2% | 2026-11-11 |
| 13 | DeepSeek-V3.2-Exp Thinking | DeepSeek | 74.2% | 2026-11-11 |
| 14 | Claude Opus 4 Thinking | Anthropic | 72.0% | 2026-11-11 |
| 15 | o4-mini High | OpenAI | 72.0% | 2026-11-11 |
| 16 | DeepSeek R1 0528 | DeepSeek | 71.4% | 2026-11-11 |
| 17 | Claude Opus 4 | Anthropic | 70.7% | 2026-11-11 |
| 18 | DeepSeek Chat | DeepSeek | 70.2% | 2026-11-11 |
| 19 | DeepSeek-V3.2-Exp | DeepSeek | 70.2% | 2026-11-11 |
| 20 | Claude Sonnet 3.7 Thinking | Anthropic | 64.9% | 2026-11-11 |
| 21 | o1 High | OpenAI | 61.7% | 2026-11-11 |
| 22 | Claude Sonnet 4 Thinking | Anthropic | 61.3% | 2026-11-11 |
| 23 | Claude Sonnet 3.7 | Anthropic | 60.4% | 2026-11-11 |
| 24 | o3-mini High | OpenAI | 60.4% | 2026-11-11 |
| 25 | Qwen3 235B-A22B | Alibaba | 59.6% | 2026-11-11 |
| 26 | Qwen3 235B-A22B Instruct 2507 | Alibaba | 59.6% | 2026-11-11 |
| 27 | Kimi K2 Instruct | Moonshot AI | 59.1% | 2026-11-11 |
| 28 | Kimi K2 Instruct 0905 | Moonshot AI | 59.1% | 2026-11-11 |
| 29 | DeepSeek-R1 | DeepSeek | 56.9% | 2026-11-11 |
| 30 | Claude Sonnet 4 | Anthropic | 56.4% | 2026-11-11 |
| 31 | DeepSeek V3 0324 | DeepSeek | 55.1% | 2026-11-11 |
| 32 | Gemini 2.5 Flash Preview 05-20 Thinking | 55.1% | 2026-11-11 | |
| 33 | o3-mini Medium | OpenAI | 53.8% | 2026-11-11 |
| 34 | Grok 3 | xAI | 53.3% | 2026-11-11 |
| 35 | GPT-4.1 | OpenAI | 52.4% | 2026-11-11 |
| 36 | Claude Sonnet 3.5 v2 | Anthropic | 51.6% | 2026-11-11 |
| 37 | Grok-3 mini High | xAI | 49.3% | 2026-11-11 |
| 38 | DeepSeek-V3 | DeepSeek | 48.4% | 2026-11-11 |
| 39 | Gemini 2.5 Flash (Apr 2025) | 47.1% | 2026-11-11 | |
| 40 | GPT-4o (Mar 2025) | OpenAI | 45.3% | 2026-11-11 |
| 41 | GPT-4.5 | OpenAI | 44.9% | 2026-11-11 |
| 42 | Gemini 2.5 Flash Preview 05-20 | 44.0% | 2026-11-11 | |
| 43 | GPT OSS 120B High | OpenAI | 41.8% | 2026-11-11 |
| 44 | Qwen3 32B | Alibaba | 40.0% | 2026-11-11 |
| 45 | Gemini 2.0 Flash (Dec 2024) | 38.2% | 2026-11-11 | |
| 46 | Gemini 2.0 Pro | 35.6% | 2026-11-11 | |
| 47 | Grok-3 mini Low | xAI | 34.7% | 2026-11-11 |
| 48 | o1-mini | OpenAI | 32.9% | 2026-11-11 |
| 49 | GPT-4.1 mini | OpenAI | 32.4% | 2026-11-11 |
| 50 | Claude Haiku 3.5 | Anthropic | 28.0% | 2026-11-11 |
| 51 | GPT-4o (Jan 2025) | OpenAI | 27.1% | 2026-11-11 |
| 52 | GPT-4o (2024-08-06) | OpenAI | 23.1% | 2026-11-11 |
| 53 | Qwen Max | Alibaba | 21.8% | 2026-11-11 |
| 54 | QwQ 32B | Alibaba | 20.9% | 2026-11-11 |
| 55 | GPT-4o (2024-11-20) | OpenAI | 18.2% | 2026-11-11 |
| 56 | Gemini 2.0 Flash Thinking (Jan 2025) | 18.2% | 2026-11-11 | |
| 57 | DeepSeek-V2.5 (Sep 2024) | DeepSeek | 17.8% | 2026-11-11 |
| 58 | Qwen2.5-Coder-32B-Instruct | Alibaba | 16.4% | 2026-11-11 |
| 59 | Llama 4 Maverick | Meta | 15.6% | 2026-11-11 |
| 60 | Yi-Lightning | 01.AI | 12.9% | 2026-11-11 |
| 61 | Cohere Command A | Cohere | 12.0% | 2026-11-11 |
| 62 | Codestral (latest) | Mistral | 11.1% | 2026-11-11 |
| 63 | openhands-lm-32b-v0.1 | Unknown | 10.2% | 2026-11-11 |
| 64 | GPT-4.1 nano | OpenAI | 8.9% | 2026-11-11 |
| 65 | Gemma 3 27B IT | 4.9% | 2026-11-11 | |
| 66 | GPT-4o mini | OpenAI | 3.6% | 2026-11-11 |