MASK Benchmark (Honesty)
A benchmark that separates honesty from accuracy, measuring whether language models lie under pressure rather than just whether they are correct.
Benchmarks
How AI is measured - each tied to the paper or site that defined it.
111 entries, all primary-sourced
ARC-AGI-2
The 2025 successor to ARC-AGI, a harder reasoning benchmark that frontier AI systems initially scored in the single digits on.
BrowseComp
OpenAI benchmark of 1,266 questions that force a web agent to dig persistently for hard-to-find facts.
HalluLens
A Meta hallucination benchmark built on a clear taxonomy, with tasks that regenerate to resist leakage.
GDPval
OpenAI benchmark scoring AI on real economically valuable work across 44 occupations in nine GDP sectors.
Long-Horizon-Terminal-Bench
A 46-task terminal benchmark with dense partial-credit grading where the best agent scores just 15.2 percent.
BusinessCaseBench: Grading Frontier AI on Business-School Case Analysis
A Wharton, CMU, Penn, and Harvard team benchmarked frontier AI on business cases across 18 disciplines; models already score highly on instructor rubrics.
CryptanalysisBench
A 191-task benchmark where five frontier models break 65 to 86 percent of already-broken cryptographic schemes.
Redwood Research and Anthropic launch the Conceptual Reasoning Index
A three-benchmark index scores models on unverifiable conceptual argument; Opus 5 leads at 73.6 against an estimated ceiling of 91.
ASI-Bench
A 60-task benchmark of project-level scientific research where agent scores fall from 50.91 to 26.62 once methodological guidance is removed.
SWE Refactor Bench: Whole-Repository Stack Migrations for Coding Agents
Twenty whole-repository migrations where only 28 of 520 frontier agent runs passed all three evaluation stages.
FrontierChallenge: Evaluating Scientific Workflow Completion
A cross-domain science benchmark where the best agent configuration finished only 20 of 97 end-to-end workflows.
The First Double-Blind Evaluation of a Frontier Model
Google DeepMind tested a Gemini model against confidential benchmarks inside a cryptographically sealed enclave.
Ai2's BenchMIRT audits what LLM benchmarks actually measure, question by question
Ai2 releases BenchMIRT, an item-response-theory tool that audits LLM benchmarks per question, trained on 100 models and 34,000-plus questions
WorldBench: A Culturally Grounded Benchmark for Multilingual Agents
1,600 sandbox tasks across seven languages and eight cultures, where frontier models reach only 49.2 percent.