These two can get used like synonyms, including by people who should know better. A model “scores well on benchmarks,” a company “is running evals,” and it sounds like one activity happening at two different volumes.
They’re separate tests built for separate reasons, and the gap between them explains how a model can lead every leaderboard and still be useless in your office.
Benchmark
A benchmark is a fixed, public test that everyone runs against everyone’s models. MMLU asks multiple choice questions across dozens of subjects. SWE-bench hands models real bug reports pulled from GitHub. GDPval scores tasks drawn from actual occupations. Same questions, same scoring, so the numbers can sit next to each other in a chart and mean something. That’s where the percentages in every model launch post come from.
That comparability is the whole point, and it’s also the problem. Benchmark questions leak into training data over time, so a rising score might mean the model improved or might mean it has seen the test. And once every model clusters near the top of one, it stops separating anything, which is why new benchmarks keep appearing.
Eval
An eval is a test you build out of your own work. Fifty support emails your team already answered correctly. Two hundred contracts where you know exactly which clause was the risky one. You run the model against them and grade what comes back against what you know is right. It’s tedious, and there’s no shortcut version of it that works.
Nobody else can use your eval, and that’s fine, because you’re not ranking the model against its competitors. You’re finding out whether it can do one specific job at a rate you can live with. Most companies doing serious AI work have a folder of these somewhere, usually built by whoever got tired of guessing.
The tell
Ask whether someone at another company could run the same test and compare notes with you. If yes, it’s a benchmark. If the test only makes sense inside your building, it’s an eval.
Worth knowing which one you’re being handed. Benchmarks are useful, and I’d rather have them than not, but they were built to compare models to each other, not to predict how one will do at your particular work. The benchmark score arrives from the company selling you the model. The eval is how you find out whether that number has anything to do with your Tuesday.