SWE-bench Verified (Epoch run)
500 human-verified GitHub issues from 12 Python repositories, run by Epoch AI with one fixed scaffold so models are comparable. Close to saturation.
| Category | Coding |
|---|---|
| Metric | Accuracy |
| Source | Epoch AI (evaluator) |
| License | CC BY 4.0 |
| Benchmark page | epoch.ai |
| Status | Scored |
| Weight | 0.8 within Coding |
| Fit | Discrimination 0.52, noise 0.19 |
| Lineage | Shares weight with swe_bench_verified |
Results
| # | Model | Provider | Result | Date |
|---|---|---|---|---|
| 1 | Claude Opus 4.7 Max | Anthropic | 83.5% | 2026-04-20 |
| 2 | GPT-5.5 Extra High | OpenAI | 80.6% | 2026-04-24 |
| 3 | Gemini 3.5 Flash High | 79.3% | 2026-06-01 | |
| 4 | Claude Opus 4.6 | Anthropic | 78.7% | 2026-02-18 |
| 5 | GLM-5.2 Max | Zhipu AI | 78.7% | 2026-06-25 |
| 6 | DeepSeek V4 Pro Max | DeepSeek | 77.6% | 2026-06-18 |
| 7 | Qwen3.7 Max | Alibaba | 77.3% | 2026-06-18 |
| 8 | GPT-5.4 High | OpenAI | 76.9% | 2026-03-06 |
| 9 | Kimi K2.6 | Moonshot AI | 76.7% | 2026-05-08 |
| 10 | Claude Opus 4.5 | Anthropic | 76.7% | 2026-02-05 |
| 11 | Qwen3.6 Max Preview | Alibaba | 76.7% | 2026-05-28 |
| 12 | Gemini 3.1 Pro Preview Custom Tools | 75.6% | 2026-02-24 | |
| 13 | Gemini 3 Flash Preview | 75.4% | 2026-02-18 | |
| 14 | Claude Sonnet 4.6 | Anthropic | 75.2% | 2026-02-21 |
| 15 | GPT-5.3 Codex High | OpenAI | 74.8% | 2026-02-25 |
| 16 | GLM-5.1 | Zhipu AI | 74.2% | 2026-05-15 |
| 17 | Kimi K2.5 | Moonshot AI | 73.8% | 2026-02-17 |
| 18 | GPT-5.2 High | OpenAI | 73.8% | 2026-02-12 |
| 19 | GPT-5 High | OpenAI | 73.6% | 2026-02-06 |
| 20 | Claude Opus 4.1 | Anthropic | 73.4% | 2026-02-11 |
| 21 | Gemini 3 Pro Preview | 72.9% | 2026-02-13 | |
| 22 | GLM-5 | Zhipu AI | 72.1% | 2026-02-15 |
| 23 | GPT-5 Medium | OpenAI | 71.5% | 2026-02-05 |
| 24 | Claude Sonnet 4.5 | Anthropic | 71.3% | 2026-02-05 |
| 25 | Claude Opus 4 | Anthropic | 70.7% | 2026-02-06 |
| 26 | GPT-5.1 High | OpenAI | 68.0% | 2026-02-18 |
| 27 | GPT-5 Mini Medium | OpenAI | 64.7% | 2026-02-01 |
| 28 | o3 Medium | OpenAI | 62.3% | 2026-02-12 |
| 29 | Claude Sonnet 3.7 | Anthropic | 61.0% | 2026-02-04 |
| 30 | Qwen3.6 Plus | Alibaba | 57.9% | 2026-05-14 |
| 31 | Gemini 2.5 Pro | 57.6% | 2026-02-13 | |
| 32 | GPT-4.1 | OpenAI | 48.5% | 2026-02-08 |
| 33 | GPT-4o (2024-11-20) | OpenAI | 31.0% | 2026-02-11 |