**SWE-bench** is a benchmark built from real GitHub issues. The agent gets a bug report plus the full source repository, and it must produce a code patch that makes the repository's existing tests pass. HumanEval is much smaller: you get a function signature and a docstring, and you write that one function from scratch.
The correct answer is the first option. SWE-bench is harder because of task structure, not surface trivia. The agent has to find which files matter inside a large codebase, understand how the existing code works, edit several files consistently, and run tools like search and tests to check itself.
HumanEval, by contrast, fits in a single prompt. There are no cross-file dependencies, no need to explore, and the spec is complete. You just synthesise one short, self-contained function.
The other three options sound plausible but miss the point. SWE-bench being Python-heavy, or whether it asks you to write tests, or whether repos are private, are not the reasons it is hard. The difficulty is the realistic, open-ended, multi-step nature of fixing a real issue in a real project.
**SWE-bench** is harder than HumanEval because of task structure, not language or access control. So the first option is correct.
Each SWE-bench task is a real GitHub issue paired with the full repository at the buggy commit. The agent must localise the relevant code, understand existing behaviour, and edit across multiple files. It then has to produce a patch that passes the repo's hidden test suite.
That demands skills HumanEval never tests. Repository navigation, long-context reasoning, tool use like grep and test execution, and not breaking unrelated functionality. HumanEval is a single self-contained function with a complete spec. It fits in one prompt with no exploration needed.
The distractors miss the mechanism. Python-only is a coverage detail, not a difficulty driver. SWE-bench does not require the agent to author tests. The tests already exist and serve as the grader. And the repos are public, which is exactly why contamination is a known concern, not a guarantee against it.
Resolution rate is the headline metric. It measures the fraction of issues where the generated patch passes every required test.
Quick reference
TL;DR
SWE-bench gives an agent a real GitHub issue and a whole repository. It must localise the bug, edit across files, and pass the repo's hidden tests. HumanEval is one self-contained function.
Imagine two cooking tests. In the first, you get a recipe card that says 'make one omelette' with every ingredient measured out on the counter. You just follow the steps. That is HumanEval: one small, fully specified task. In the second test, someone hands you a note that says 'the restaurant's signature dish tastes wrong lately, fix it', and points at the entire kitchen. You have to figure out which dish, taste it, search the pantry, find the bad ingredient hidden among hundreds, change the recipe, and then the food critic decides whether it passes. That is SWE-bench: a real bug report in a real codebase. The hard part is not writing code, it is finding where the problem lives across thousands of files and not breaking everything else.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Evaluating a base model with no agent scaffold, tools, or repository access SWE-bench assumes navigation and test execution. Without tools the score reflects scaffolding gaps, not model skill.
Measuring isolated algorithm or single-function synthesis ability HumanEval or LiveCodeBench fit better. SWE-bench mixes in localisation and editing that blur the signal you want.
Comparing models trained before and after a fixed date on the public set Contamination confounds the comparison. Use a time-sliced or held-out variant so neither model could memorise the fix.
Judging code quality, readability, or maintainability of the patch Test-pass is binary and overfit prone. A patch can pass and still be unidiomatic, so it needs human or static review.
Quick scan · 2–4 min
Everything important, quickly.
5 min: why repository-level structure is the difficulty driver, the four-stage decomposition, what resolution rate means, the contamination concern, and why SWE-bench Verified exists.
Real-world examples
Real products, models, and research that use this idea.
1SWE-bench Verified is the human-filtered subset most 2026 model cards report, used because the raw set contains underspecified or unsolvable tasks.
2Frontier agent systems built on Claude Opus 4.7 and GPT-5.5 report SWE-bench Verified resolution rates as a headline coding capability number.
3OpenHands and SWE-agent are open scaffolds that give a model file-edit and shell tools so it can navigate the repo and run tests.
4Time-sliced variants drawn from post-cutoff GitHub issues are used to probe whether high scores reflect memorisation.
5LangSmith and similar eval platforms log per-step agent trajectories so teams can see where localisation or editing failed, not just the final pass or fail.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you tell whether a high SWE-bench score reflects reasoning or pretraining contamination?
Sign in or go Pro to see the approach for this follow-up.
QWhy does SWE-bench report resolution rate instead of a pass@k style metric like HumanEval?
Sign in or go Pro to see the approach for this follow-up.
QWhat does SWE-bench Verified change, and why was it introduced?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying SWE-bench is harder because it is Python only or uses private repos. The real driver is task structure: repo navigation, multi-file edits, tool use, and passing hidden tests.