☀️ AI Morning Minute: Parameters
When someone says a model has 70 billion parameters, they’re not being vague. That number means something specific, and it explains a lot about what the model can and can’t do.
You’ve seen the numbers on AI model announcements: 7 billion parameters, 70 billion, 405 billion. They sound impressive, and they’re meant to. But parameters aren’t just a marketing metric. They’re the actual numerical values the model learned during training, the ones that determine every response it gives. Understanding what they are makes a lot of other AI concepts click into place.
What it means
Parameters are the numerical values stored inside a neural network that encode what the model has learned. Think of them as billions of tiny dials, each set to a specific value during training. When you send a prompt to an AI model, those dials are what transform your input into an output. The model doesn’t retrieve answers from a database. It runs your text through a mathematical process involving all those values, and the result is the response.
During training, backpropagation adjusts each parameter slightly, again and again, across billions of examples, until the values converge on settings that produce useful outputs. After training, the parameters are frozen. What you’re interacting with when you use ChatGPT or Claude is a fixed set of numbers doing very fast math.
Parameter count is a rough proxy for a model’s capacity. More parameters generally means more ability to store nuanced relationships between concepts. GPT-4 is estimated at around 1.8 trillion parameters. Llama 3’s 70B model has 70 billion. A model you can run on your phone might have 3 billion. Smaller doesn’t mean useless: a well-trained 7B model beats a poorly trained 70B model on most tasks.
Why it matters
Parameter count drives hardware requirements. Every parameter takes up memory. A 70 billion parameter model in standard precision needs roughly 140 gigabytes of GPU memory just to load, before you do anything with it. This is why running large models requires specialized hardware and why open-weights models are often released in quantized (compressed) versions that reduce memory footprint at some cost to quality.
The race to count parameters has largely ended. For years, bigger meant better and labs competed on raw size. That changed around 2023, when research showed that smaller models trained on more data often outperform larger models trained on less. Meta’s Llama models and Google’s Gemma line both demonstrated that a well-trained 7B or 8B model could handle most everyday tasks as well as much larger models from a year prior.
Parameters are what you download when you download an open-weights model. The weights file is the parameters file. When Llama or Mistral releases a model publicly, what they’re releasing is a set of numbers, billions of them, that encode everything the model learned. The training code and data may not be included. The parameters are the model.
Simple example
A piano has 88 keys. A skilled pianist can produce an enormous range of music from those keys, because they’ve learned which combinations, in which order, at which timing, produce which sounds. The keys are fixed. The skill is in knowing the values.
An AI model’s parameters work the same way. The architecture (the piano) is fixed after training. The parameters (the learned values) are what determine every output. Change the parameters and you have a different model, even if the architecture is identical.