SWE-bench gives an agent a real GitHub issue and a whole repository. It must localise the bug, edit across files, and pass the repo's hidden tests. HumanEval is one self-contained function.
Imagine two cooking tests. In the first, you get a recipe card that says 'make one omelette' with every ingredient measured out on the counter. You just follow the steps. That is HumanEval: one small, fully specified task. In the second test, someone hands you a note that says 'the restaurant's signature dish tastes wrong lately, fix it', and points at the entire kitchen. You have to figure out which dish, taste it, search the pantry, find the bad ingredient hidden among hundreds, change the recipe, and then the food critic decides whether it passes. That is SWE-bench: a real bug report in a real codebase. The hard part is not writing code, it is finding where the problem lives across thousands of files and not breaking everything else.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
SWE-bench asks a fundamentally different question than HumanEval. HumanEval hands the model a function signature and a docstring and asks it to synthesise that one function. SWE-bench hands the agent a real GitHub issue plus the entire repository at the commit where the bug exists, and asks it to produce a patch that makes the project's own test suite pass.
That shift, from isolated synthesis to repository-level repair, is the whole story behind the question. It is why the first option is correct and why the other three, which point at language coverage, test authorship, and repository privacy, all miss the mechanism.
The reason this matters in an interview is that benchmarks are how the field talks about progress, and conflating two very different benchmarks leads to bad decisions. A team that picks a model on its HumanEval number and then deploys it as a coding agent is measuring the wrong thing. This deep dive walks the difficulty stage by stage, defines what the headline resolution-rate metric actually measures, explains why the agent scaffold and retrieval setup move the number as much as the model does, and covers the contamination concern that makes senior interviewers probe how a SWE-bench score was produced before they trust it.
What SWE-bench actually is
SWE-bench is a benchmark of real software-engineering tasks scraped from the public issue and pull-request history of popular open-source Python projects like Django, scikit-learn, and SymPy. Each task instance bundles three things: the issue text describing a bug or feature request, a snapshot of the repository at the commit just before the fix, and a set of tests that encode the expected behaviour.
The agent never sees the tests. It reads the issue, explores the codebase, and emits a patch. The harness then applies the patch and runs two test groups. The fail-to-pass tests are the ones that should now succeed because the bug is fixed. The pass-to-pass tests are the ones that must keep succeeding so the agent has not broken anything else.
Contrast this with HumanEval, which is 164 small, hand-written programming problems. Each gives a complete specification and asks for one short, self-contained function. There is no repository, no exploration, and no cross-file state. The two benchmarks sit at opposite ends of the realism axis.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- SWE-bench Verified is the human-filtered subset most 2026 model cards report, used because the raw set contains underspecified or unsolvable tasks.
- Frontier agent systems built on Claude Opus 4.7 and GPT-5.5 report SWE-bench Verified resolution rates as a headline coding capability number.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you tell whether a high SWE-bench score reflects reasoning or pretraining contamination?
Compare scores on time-sliced tasks created after the model's training cutoff against older tasks. A large gap suggests memorisation. Also probe whether the model reproduces the exact upstream fix commit versus an independent valid patch.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying SWE-bench is harder because it is Python only or uses private repos. The real driver is task structure: repo navigation, multi-file edits, tool use, and passing hidden tests.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.