SWE-bench, Terminal-Bench & Co.: Reading Coding-Agent Benchmarks

SWE-bench tests whether an agent can fix real GitHub issues with a passing patch. Terminal-Bench tests broader terminal work. Here's what the numbers actually mean.

What SWE-bench actually measures

SWE-bench takes real, resolved GitHub issues from open-source Python repositories, paired with the pull request that fixed them. An agent gets the codebase and issue, and produces a patch, graded automatically: the test suite runs, and the fix counts as correct only if previously failing tests now pass, without breaking tests that already passed.

Why SWE-bench Verified exists

The original SWE-bench has 2,294 tasks, but not all were reliably solvable or fairly graded - some had flaky tests or vague descriptions. OpenAI had 93 Python developers manually screen 1,699 tasks, keeping the 500 confirmed solvable as SWE-bench Verified. GPT-4o's best score on that cleaner set more than doubled versus the original.

Terminal-Bench: broader than code fixes

Terminal-Bench, a Stanford and Anthropic collaboration, tests agents on wider terminal tasks beyond fixing code - sysadmin, security, data science, ML setup, like building a Linux kernel or configuring a git server. It measures how an agent handles the terminal itself.

Reading these numbers without getting fooled

A SWE-bench score tells you about GitHub-issue-shaped, Python-heavy, test-verifiable tasks, not every kind of coding work. For how to read benchmark numbers in general, see the chapter "Reading Benchmarks Without Getting Fooled".

EXAMPLE

Reading a claim like 'Model X scores 70% on SWE-bench Verified': that means it produced a correctly passing patch for roughly 350 of the 500 human-verified, solvable GitHub issues in that set - not 70% of coding tasks in general.

🛠️ EXERCISE — TRY IT YOURSELF

Look up a real SWE-bench Verified or Terminal-Bench leaderboard entry and read it critically instead of just as a ranking.

Open swebench.com or tbench.ai and pick a model with a high placement.
Find out how many tasks the benchmark contains and what kind of tasks they are.
Think about which properties of your own project, language, size, age of the codebase, don't appear in the benchmark.
Write one sentence describing what the score actually tells you, and what it doesn't.

✅ SELF-CHECK

☐ Can you explain where SWE-bench's tasks come from and how they're scored automatically?
☐ Do you know why SWE-bench Verified is considered more reliable than the original dataset?
☐ Did you name one concrete difference between the benchmark tasks and your own project?

QUICK QUIZ

Why was SWE-bench Verified created as a separate, smaller subset of the original SWE-bench?