Evaluation — benchmarks, LM-as-judge, evals-as-code
**Knowledge / reasoning:** • **MMLU / MMLU-Pro** — 57 subjects, undergrad-level. Saturating at the frontier (>90%). • **GPQA (Diamond)** — PhD-level science; human experts ~65%. • **BIG-Bench Hard, ARC-AGI-1/2** — reasoning; ARC-2 human >80% vs. frontier models trailing.
**Math:** GSM8K (grade-school word problems), MATH, AIME, FrontierMath (research-grade).
**Code:** HumanEval (saturated), MBPP, LiveCodeBench, SWE-bench / SWE-bench Verified (real GitHub issues). SWE-bench Verified human-filtered subset is the primary track; RE-bench (METR) tests long-horizon ML research.
**Agents/tools:** τ-bench (airline/retail customer service), OSWorld (computer use), AgentBench, WebArena.
**Long-context:** Needle-in-Haystack (retrieval), RULER (13 task types up to 128K+), LongBench.
**Safety / harms:** MedHELM (healthcare), WMDP (weapons unlearning), TruthfulQA, HarmBench, AILuminate (MLCommons), AIR-Bench.
**LM-as-judge** — use a strong model to score outputs against a rubric. Fast, cheap, scales. **Failure modes:** position bias, length bias, self-preference. Mitigations: randomize order, pairwise rather than absolute, use diverse judges, and correlate with human eval periodically.
**Evals-as-code:** treat evals like unit tests. Version the prompts, datasets, and rubric; track regressions in CI. Anthropic's Anthropic Workbench, OpenAI Evals, LangSmith, Arize Phoenix, Braintrust, Galileo, Weights & Biases Weave support this workflow.
**Contamination** — frontier pretraining data ingests everything; published benchmarks risk being in the training set. Mitigations: held-out test sets (SWE-bench Verified 'canaries'), continuously refreshed evals, private evals, and meta-benchmarks like MixEval.