Most software gets tested the way you’d test a ladder. You climb it a bunch of times, and if it doesn’t break, you call it safe. Formal verification is the other approach: proving, step by step, that it can’t break under the conditions you care about.
It’s been around for decades and it’s always been brutally slow, which is why almost nobody bothers. That’s starting to change, because AI turns out to be pretty good at the tedious part.
What it means
Formal verification means writing a proof (in a special language a computer can read) that a program or a math argument follows its rules. Then a piece of software called a proof checker confirms every single step. Lean and Rocq are two common ones.
The checker doesn’t take anyone’s word for anything. If even one step doesn’t follow, the whole proof fails. That’s what makes it a nice match for AI. Models still have hallucinations, but a made-up step doesn’t get past the checker.
Why it matters
AI can now do the grunt work. Anthropic and a company called Theorem had Claude write about 180 million lines of proofs covering 239 entry points across 123 open-source libraries. Along the way, it rediscovered a known bug in a widely used library called libexpat. (It was already on record, so this wasn’t a new scare.)
It works on math too. The startup Axiom says its AI agents rewrote one of the biggest proofs in math, the roughly 15,000-page classification of finite simple groups (originally written by about 100 mathematicians), in Lean in 24 days. It turned up about 34 problems, and one needed brand new math to fix.
A proof is only as good as the question it answers. Anthropic’s proofs cover certain kinds of failures, on one type of processor setup, with inputs capped at about 64 MB. A proof can be perfect and still prove the wrong thing, so somebody still has to decide what “correct” means.
Simple example
A town wants to know if the old bridge is safe. One way to find out: drive heavier and heavier trucks over it and see if anything cracks. That catches a lot. But it only tells you about the trucks you actually tried.
The other way: an engineer runs the numbers on every beam and proves the bridge holds any truck under 20 tons. Nobody has to test every truck. And that 20-ton limit on the sign is the part people forget to read.

