DeepSWE v1.1
2026-09-22 · General long-horizon engineering
- What it measures
- General long-horizon engineering: feature-sized changes in real open-source repos. 113 tasks across Go, Python, TypeScript, JavaScript, Rust, and more, spanning 91 active repositories. The agent commits in an isolated environment and is graded in a pristine container. On this site it is a general board and appears in the Web frontend, Android, and CLI / DevOps scenarios.
- How
- Datacurve's DeepSWE v1.1. Tasks are written from scratch, not adapted from existing commits or PRs. All public scores use Pier with mini-swe-agent. This site follows the official Best view: one row per model at its highest Pass@1 reasoning tier; ties break toward higher effort.
- What the score means
- The percent is Pass@1: attempt pass rate over scored rollout attempts. Context-window failures and agent timeouts count as failures; provider, verifier, and network errors are excluded. Higher is better.
- Caveats
- The public Best chart hides some older models; this page keeps all 28 models at their Best config. Every row uses mini-swe-agent, so native CLIs such as Claude Code or Codex are not compared. The cost column is mean per-task cost from the source JSON; the site UI may round costs differently.
- Updates
- The source board is marked updated 2026-09-22; artifact generated_at is 2026-09-22T06:27:15Z. This page synced 28 Best-config rows from https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json on 2026-10-11.
Full board
28
| # | Model | Score |
|---|
| 1 | GPT 6 Astraxhigh | 74.1 |
| 2 | Gemini 3.8 Flashhigh | 73.8 |
| 3 | Claude Opus 5max | 73.6 |
| 4 | GPT 5.6 Solmax | 72.7 |
| 5 | Claude Fable 5xhigh | 69.9 |
| 6 | GPT 5.6 Terramax | 69.6 |
| 7 | GLM 5.3max | 69.0 |
| 8 | Kimi K3max | 68.5 |
| 9 | Grok 4.6medium | 67.5 |
| 10 | GPT 5.6 Lunamax | 67.2 |
| 11 | GPT 5.5xhigh | 67.0 |
| 12 | Gemini 3.7 Flashmedium | 65.5 |
| 13 | GLM 5.3 Flashmax | 63.4 |
| 14 | DeepSeek V4 Promax | 62.8 |
| 15 | Claude Opus 4.8max | 59.0 |
| 16 | Qwen 3.8 Maxxhigh | 57.5 |
| 17 | Muse Spark 1.2xhigh | 54.9 |
| 18 | Claude Sonnet 5max | 53.8 |
| 19 | Grok 4.5high | 53.8 |
| 20 | DeepSeek V4 Flashmax | 53.3 |