NL2Repo-Bench evaluates whether coding agents can build a complete, installable Python repository from a natural-language specification and an empty workspace. The 104 tasks are graded by the average pass rate of the upstream pytest suites.
Browse the latest scores, model modes, release dates, and parameter sizes for NL2Repo-Bench.
Data sourced primarily from official releases (GitHub, Hugging Face, papers), then benchmark leaderboards, then third-party evaluators. Learn about our data methodology
| Rank | Model | License | |||
|---|---|---|---|---|---|
— |  Claude Opus 5 Thinking Level · MaxTools | 75.30 | 2026-07-24 | Unknown | Closed |
— |  DeepSeek-V4.1-Flash Thinking Level · MaxTools | 65.40 | 2026-09-10 | 552B | Free Commercial |
— |  Qwen3.8-Max-0902 Thinking Level · Extra HighTools | 64.90 | 2026-09-02 | 2400B | Closed |
— | DeepSeek-V4-Pro Thinking Level · Extra HighTools | 61.50 | 2026-08-13 | 1600B | Free Commercial |
— |  Hy4 preview Thinking Level · HighTools | 58.90 | 2026-08-28 | 770B | Free Commercial |
— |  Kimi K3 Thinking Level · MaxTools | 58.00 | 2026-07-16 | 2800B | Conditional |
— |  GLM-5.3 Thinking Level · MaxTools | 58.00 | 2026-08-14 | 744B | Conditional |
— | DeepSeek-V4-Flash-Vision-Exp Thinking Level · MaxTools | 57.70 | 2026-08-21 | 305B | Free Commercial |
— |  GPT-5.6 Sol Thinking Level · MaxTools | 56.80 | 2026-06-26 | Unknown | Closed |
— | GLM-5.3-Flash Thinking Level · MaxTools | 56.30 | 2026-08-26 | 320B | Free Commercial |
— | Qwen3.8-Max Thinking Level · Extra HighTools | 55.90 | 2026-08-03 | 2400B | Conditional |
— | DeepSeek-V4-Flash Thinking Level · MaxTools | 54.20 | 2026-04-24 | 284B | Free Commercial |
— | GLM-5.2 Thinking EnabledTools | 48.90 | 2026-06-13 | 753.3B | Free Commercial |
— | Qwen3.8-Flash-Next Thinking Level · Extra HighTools | 48.10 | 2026-08-26 | 125B | Conditional |
— | Qwen3.8-27B Thinking EnabledTools | 42.30 | 2026-08-14 | 27B | Free Commercial |
— |  MiniMax-M2.7 Thinking EnabledTools | 39.80 | 2026-03-18 | 229B | Non-Commercial |
