☀️ AI Morning Minute: Benchmark
Every AI company says their model is the best. Benchmarks are how you check.
When a lab releases a new model, the announcement usually comes with a set of scores: 87% on this test, 94% on that one, state of the art on another. Those scores come from benchmarks. Without them, AI capability claims would be impossible to compare or verify. With them, you can at least ask whether the test being cited actually measures something that matters, and whether the model was trained to pass the test or to genuinely understand the task.
What it means
A benchmark is a standardized test used to measure and compare AI model performance. Benchmarks typically consist of a dataset of questions, problems, or tasks with known correct answers, and a scoring method that lets you compare results across different models. They cover a wide range of capabilities: reasoning, coding, math, language understanding, safety, and increasingly, real-world task completion. Well-known examples include MMLU (a test of knowledge across 57 academic subjects), HumanEval (coding ability), and ARC-AGI-3 (abstract reasoning designed to resist pattern-matching). Benchmarks are run by labs, third parties, and independent researchers, and results are often published on public leaderboards.
Why it matters
Benchmarks are the primary language of AI progress. When a lab says its new model is better, they mean better on benchmarks. Understanding what a benchmark measures, and what it doesn’t, is the difference between reading an AI announcement critically and taking it at face value.
Benchmark saturation is a real problem. When frontier models score above 90% on a test, that test stops being useful for distinguishing between them. The field has burned through dozens of benchmarks this way. New ones get designed, models get trained on overlapping data, scores climb fast, and the cycle repeats. This pattern is called benchmark contamination.
The hardest benchmarks are now the most important ones. ARC-AGI-3, Terminal-Bench, and similar tests are designed to measure capabilities that can’t be gamed by memorization. They’re harder to score well on and harder to contaminate. That’s why labs treat high scores on them as meaningful signals rather than marketing.
Simple example
Think of benchmarks the way you’d think of standardized tests in education. A student who scores 98% on a multiple choice test might have genuinely mastered the material, or might have been coached on that test’s question patterns.
The score looks the same either way. AI benchmarks have exactly this problem, which is why the field keeps designing new ones that are harder to teach to.
And now for something not AI related...
Have I mentioned PostARTing — an art subscription that mails original, handmade art postcards to your door? It’s a new project that my wife and I are launching and we think you are gonna like it.
So, we’re opening a limited Charter Club before we officially launch. Founding pricing locked in forever, plus a free card to send a friend.
If getting real art in the mail sounds like your kind of thing, claim a spot before they’re all gone.