☀️ AI Morning Minute: Tokens vs. Parameters
Two numbers in every model announcement, and almost everyone gets them mixed up.
The confusion
Every model launch comes with big numbers attached. Trained on 15 trillion tokens. Has 70 billion parameters. Both sound like measurements of size, so people treat them as interchangeable. Buuut they’re not even the same kind of thing.
One is what the model ate.
The other is what the model became.
The difference
Tokens are pieces of text, roughly three-quarters of a word each. When a company says a model trained on 15 trillion tokens, they’re describing the reading list: how much text the model saw during training. Tokens also measure your conversations. Every prompt you type gets chopped into tokens, counted, and (if you’re paying for API access) billed.
Parameters are the knobs inside the model. Each one is a number the model adjusted during training to get better at predicting text. A 70 billion parameter model has 70 billion of those knobs. When training ends, the tokens are gone. The model doesn’t store the text it read. What remains is the pattern of all those knob settings, which is the model itself.
So tokens flow through the model twice: once as training data, then forever as your prompts and its replies. Parameters stay put. They’re the machine the tokens pass through.
The confusion gets expensive when people reason from the wrong number. A model with more parameters isn’t automatically smarter, and a model trained on more tokens isn’t automatically better. The ratio between them matters more than either number alone. Researchers found that many early models were oversized: too many parameters, not enough tokens to justify them. Smaller models trained on more text often win.
The one-sentence test
If the number describes text going in or out, it’s tokens. If the number describes the model itself, it’s parameters.
Simple example
A chef spends ten years working through 15,000 recipes. The recipes are the tokens: the raw material, consumed and gone. Nobody asks the chef to recite recipe 4,302, and she couldn’t. What the ten years built is her judgment: when to salt, how hot the pan should be, what “done” looks like. That judgment is the parameters.
You can’t point to where a single recipe lives in her hands, and you can’t find a single sentence stored in a model’s weights. The recipes made the chef. They aren’t in the chef.