Convex Evals
2026-10-08 · Framework eval · 92a60e0bb2f156b5ac44e6ea3f5d03a91deb434b9d002d14819d1fc2b6e35cdb
- What it measures
- Models write Convex code and the harness runs it. The current version has 112 tasks. The baseline is No Guidelines: no supplied guidelines and no web access.
- How
- The official Convex coding evals. Rank uses only No Guidelines. No Guidelines + Web is another column: the model may search and open pages, still without guidelines. The columns are not averaged or added.
- What the score means
- The percent is the pass rate on 112 tasks, kept to two decimals. The web column appears only when that model has a run. A missing run is not a zero.
- Caveats
- Convex writes the tasks and grades them. This measures a model together with Convex, not one framework against another. The front is near the ceiling. With Guidelines is omitted: on 2026-10-10 the public query runs:leaderboardScores with experiment=with_guidelines returned a server error. The With Guidelines tab has no experiment field. The unscoped default query is not treated as that arm.
- Updates
- Version 92a60e0bb2f156b5ac44e6ea3f5d03a91deb434b9d002d14819d1fc2b6e35cdb, minted 2026-10-08. This page synced No Guidelines (13 models) and No Guidelines + Web (8 models) from fabulous-panther-525.convex.cloud on 2026-10-10.
Baseline board
| # | Model | Baseline | No guidelines + web | Cost per run |
|---|---|---|---|---|
| 1 | Claude Opus 5.5 1 run | 90.18 | 96.43 | $3.64 |
| 2 | GPT-6.1 Sol 1 run | 88.39 | 91.96 | $1.21 |
| 3 | Claude Sonnet 5.5 1 run | 84.82 | 84.82 | $1.42 |
| 4 | Claude Opus 5 1 run | 80.36 | 83.93 | $4.53 |
| 5 | Grok 4.7 1 run | 80.36 | — | $4.65 |
| 6 | Claude Haiku 5.5 2 runs | 74.55 | 79.02 | $0.16 |
| 7 | Gemini 3.1 Pro Preview 1 run | 72.32 | — | $3.99 |
| 8 | GPT 6 Luna 1 run | 67.86 | 67.86 | $0.08 |
| 9 | DeepSeek V4.1 Flash 1 run | 66.07 | 75.0 | $0.16 |
| 10 | Gemini 3.8 Flash 1 run | 64.29 | — | $2.40 |
| 11 | MiMo V2.6 Pro 1 run | 54.46 | — | $0.40 |
| 12 | MiMo V2.6 Flash 1 run | 48.21 | — | $0.10 |
| 13 | GLM-5.3 1 run | 42.86 | 39.29 | $0.45 |