SWE-bench gives an agent a real GitHub issue and a whole repository. It must localise the bug, edit across files, and pass the repo's hidden tests. HumanEval is one self-contained function.
Imagine two cooking tests. In the first, you get a recipe card that says 'make one omelette' with every ingredient measured out on the counter. You just follow the steps. That is HumanEval: one small, fully specified task. In the second test, someone hands you a note that says 'the restaurant's signature dish tastes wrong lately, fix it', and points at the entire kitchen. You have to figure out which dish, taste it, search the pantry, find the bad ingredient hidden among hundreds, change the recipe, and then the food critic decides whether it passes. That is SWE-bench: a real bug report in a real codebase. The hard part is not writing code, it is finding where the problem lives across thousands of files and not breaking everything else.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: why repository-level structure is the difficulty driver, the four-stage decomposition, what resolution rate means, the contamination concern, and why SWE-bench Verified exists.
Real products, models, and research that use this idea.
- SWE-bench Verified is the human-filtered subset most 2026 model cards report, used because the raw set contains underspecified or unsolvable tasks.
- Frontier agent systems built on Claude Opus 4.7 and GPT-5.5 report SWE-bench Verified resolution rates as a headline coding capability number.
- OpenHands and SWE-agent are open scaffolds that give a model file-edit and shell tools so it can navigate the repo and run tests.
- Time-sliced variants drawn from post-cutoff GitHub issues are used to probe whether high scores reflect memorisation.
- LangSmith and similar eval platforms log per-step agent trajectories so teams can see where localisation or editing failed, not just the final pass or fail.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you tell whether a high SWE-bench score reflects reasoning or pretraining contamination?
QWhy does SWE-bench report resolution rate instead of a pass@k style metric like HumanEval?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying SWE-bench is harder because it is Python only or uses private repos. The real driver is task structure: repo navigation, multi-file edits, tool use, and passing hidden tests.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.