SWE-bench
A benchmark of 2,294 real GitHub issues and their fixes—measuring how well models can solve actual software engineering tasks.
Think of it like
Like a real engineering job interview where you have to fix bugs in an open-source project.
Example
Given a GitHub issue and a codebase, the model has to write a patch that fixes the problem and passes all tests.
How it actually works
SWE-bench is harder than HumanEval because it requires understanding large codebases, reading error messages, iterating, and integrating fixes into existing systems. It's frontier-hard (top models solve ~10–20% of issues). It's valuable because it measures what actually matters to engineers.
For product teams
SWE-bench is emerging as the key benchmark for AI coding assistants. Models that score well here are production-ready.
For engineers
Evaluate on both 'lite' (simpler issues) and 'full' (harder issues) versions; check solution quality, not just pass rate.
Related
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome