So yeah, somebody had to justify the money. Before anyone spends nine figures on a training run, they need a reason to believe the resulting model will be better than the last one, and roughly by how much.
Scaling laws are that reason. They’re the observed relationship between how much you put into training a model and how good it comes out, and the striking part is that it holds steadily enough to forecast.
What it means
A scaling law says model performance improves in a predictable curve as you increase three things: the size of the model, the amount of training data, and the compute you spend. Researchers at OpenAI published the first influential version in 2020. Two years later a DeepMind team found the recipe was off, that models were being built too large and fed too little, and their smaller, better-fed model beat one four times its size.
Why it matters
The predictability is the whole product. A lab can run small, cheap experiments, fit a curve, and estimate what a much larger model will do before committing to build it. Without that, every training run is a coin flip with a data center attached.
The curve bends the wrong way for your wallet. These are power laws, so each equal improvement costs multiplicatively more than the one before. Going from bad to decent is cheap. Going from very good to slightly better burns through a quarter that would have funded a university.
The pretraining version may be running out of room, mostly because high quality text is finite and the labs have largely used it. That’s a real disagreement in the field, not a settled fact. Attention has shifted toward scaling how long a model thinks at answer time instead, which is a different curve with its own ceiling nobody has found yet.
Simple example
A player who lifts twice a week gets noticeably stronger by month three. Doubling to four days a week helps, though not twice as much. Going to six days a week mostly produces fatigue.
And the fix at that point usually isn’t more hours in the gym. It’s sleeping properly and eating enough, which is roughly what DeepMind figured out about training data, several billion dollars into the industry doing it the other way.