Benchmarks

SWE-bench (+Verified/Pro)

Tests whether an AI agent can resolve real GitHub issues by writing a patch that passes the repository's hidden test suite.

Originally Princeton & Stanford; the Verified subset was curated with OpenAI, the Pro variant is run by Scale AI.

⚠️ Independent audits found solutions that were effectively given away in the issue thread, and tests too weak to catch a broken fix — and SWE-bench Pro alone has several conflicting "official" scores depending on which scaffold and data split is used.

Terminal-Bench

Tests whether an agent can complete real multi-step terminal tasks, where every command's output changes what it should do next.

A Stanford x Laude (Anthropic-adjacent) collaboration, part of the open "Harbor" framework.

Aider Polyglot

Tests whether a model follows instructions and edits code correctly, unassisted, across coding exercises in several languages.

Run by Paul Gauthier, Aider's creator — not a lab or consortium.

⚠️ This leaderboard isn't re-run for every new model — it can show outdated models on top for months, simply because newer ones haven't been tested yet.

LiveCodeBench

Tests code generation, self-repair, predicting test output, and code execution, using competitive-programming problems pulled in continuously so old problems can't leak into training data.

Researchers at UC Berkeley, MIT, and Cornell (lead: Naman Jain).

LMArena Code Arena / WebDev Arena

Measures human preference, not correctness: real people blind-vote on which of two models' output (code or web-app) they like better.

Arena.ai (the LMArena / LMSYS project).

⚠️ A large-scale analysis of vendor submissions found that selectively submitting only the best-performing variant can meaningfully inflate a model's score.

HumanEval

Tests functional correctness on hand-written Python programming problems by actually running the generated code against unit tests.

Created by OpenAI researchers (Chen, Tworek, Jun and others).

⚠️ Saturated: nearly all frontier models cluster tightly together at the top, so the benchmark barely tells them apart anymore. It's used only as a basic sanity check now, not for ranking frontier models.

Artificial Analysis Coding Agent Index

A composite score for end-to-end coding-agent performance — combining several underlying benchmarks plus cost, token, and time efficiency into one index.

Run by Artificial Analysis, an independent benchmarking outfit.

⚠️ Scores the model-plus-harness combination together, so the same underlying model can rank very differently depending on which agent scaffold runs it.

ARC-AGI-2

Tests abstract, fluid reasoning through visual grid-transformation puzzles, plus how efficiently (at what cost) a model solves them. Not a coding benchmark per se.

Run by ARC Prize Inc., a nonprofit (keeps test data private or semi-private to resist contamination).

MMLU / MMLU-Pro

Tests general knowledge and reasoning with multiple-choice questions spanning many academic subjects. Not coding-specific.

Originally created by Hendrycks and colleagues (UC Berkeley-affiliated researchers).

⚠️ Saturated: frontier models cluster too closely together to differentiate. MMLU-Pro was built to fix that, but is reported to be starting to saturate too.

METR "Time Horizon"

Tracks how long a task (in human-expert work-time) an agent can complete on its own at a fixed reliability bar — read as a trend line over time, not a single score.

Run by METR, an independent AI-evaluation nonprofit.

⚠️ METR itself flags its measurements for very long tasks as unreliable with the current task suite — read it as a trend line, not a precise cutoff.

Frontier-Bench

Tests AI agents on a broad, deliberately difficult, continuously expanding set of tasks drawn from real agentic work (including coding tasks), run inside a sandbox environment — a constant supply of new tasks aims to keep the benchmark from being contaminated by training or becoming saturated.

Successor to Terminal-Bench, run by the same team behind the open "Harbor" framework (Laude Institute, Stanford-adjacent).

⚠️ A very young, constantly evolving dataset — because new tasks are added continuously, results from different points in time aren't directly comparable. The leaderboard also scores the model and agent harness (e.g. Claude Code, Codex, mini-SWE-agent) together as a pair — the same model can perform very differently depending on the scaffold.

GDPval-AA

Tests AI models on realistic knowledge-work tasks spanning numerous occupations and industries (such as legal briefs, technical drawings, or financial documents) — building on OpenAI's GDPval dataset. Models get shell and web access inside an agentic loop; an LLM judge compares two anonymized model outputs head-to-head, producing an Elo rating anchored to human expert performance.

Run by Artificial Analysis, building on the GDPval task set OpenAI developed together with industry experts.

⚠️ The evaluation method differs from OpenAI's own GDPval study: there, human industry experts judged the work; here, an LLM compares anonymized pairs — so the results aren't directly comparable to OpenAI's own GDPval numbers. It's also single-shot tasks with no rounds of revision, which by OpenAI's own account only partially captures real knowledge work.

GPQA / GPQA Diamond

Graduate-level multiple-choice questions in biology, chemistry, and physics, written and vetted by subject-matter experts — hard enough that even skilled non-experts with unrestricted internet access mostly fail them (hence "Google-proof"). "GPQA Diamond" is a smaller, more strictly filtered subset where both expert reviewers independently found the correct answer and most non-experts got it wrong — this harder variant is what leaderboards usually cite, often without saying so explicitly.

Created by a research team led by David Rein and Samuel Bowman, mostly based at New York University.

⚠️ Providers often just say "GPQA" without specifying whether they mean the full set or the smaller, harder Diamond subset — the numbers then aren't comparable. Because the dataset (with solutions) is publicly available, the authors embed a canary string to at least detect contamination from model training — it can't rule contamination out.

BrowseComp

Tests whether an AI agent can track down facts on the open web that are hard to find but clearly verifiable: short questions with exactly one correct, short answer, deliberately constructed so a simple search isn't enough — verified by the fact that the strongest models available at creation time couldn't solve them.

Published by OpenAI, with the dataset open in the simple-evals repository.

⚠️ OpenAI itself notes that the focus on short, unambiguous answers leaves it unclear how well the result transfers to real, open-ended research queries — the benchmark doesn't cover long, ambiguous, or free-text tasks. Even the human testers who worked the questions during verification could only solve a small fraction of them.