☀️ AI Morning Minute: Distillation Attack
How someone can copy a $100 million AI model for about the price of dinner.
A distillation attack is a way of stealing an AI model without ever touching its code. The attacker just talks to it, a lot, and copies what comes back. You ask the model thousands of questions, save all its answers, and use those answers to train a cheap knockoff that behaves almost the same. It’s the shadier relative of ordinary Model Distillation, which is a legitimate technique for shrinking models. Same basic move, except here nobody asked permission.
What it means
Okay, so the thing that makes this possible is that a model has to answer you to be useful. That’s the whole product. And every answer it gives away is a little piece of what it knows. Ask enough questions across enough topics, and the pile of answers you collect starts to describe the model itself.
An attacker takes that pile and uses it to fine-tune a smaller, open model they already have. They’re not copying the original’s internal wiring, which they can’t see anyway. They’re copying its behavior, which is the part that actually matters to a customer. The result is a cheaper model that does most of what the expensive one does.
And it’s cheap. In one published example, researchers copied a useful chunk of a commercial model’s ability for about $50 in usage fees. Another team cloned a specialized medical model to 94% of its performance for under $1,000. Compare that to the tens or hundreds of millions it costs to build a frontier model from scratch, and you can see why companies worry about this.
The uncomfortable part is that you can’t fully stop it. A model that refuses to answer is useless, so the answers have to flow, and if they flow, they can be collected. Defenders mostly try to make stealing more expensive than just paying: rate limits, watching for accounts that ask suspiciously broad questions, and not handing out extra detail in the responses.
Why it matters
It turns a company’s biggest investment into something copyable. The model is the asset, and the asset walks out through the front door one answer at a time.
It’s cheap enough for almost anyone. Fifty dollars and some patience is a very different threat than needing a data center.
It shapes what these companies will show you. Some of the limits you bump into exist partly to make this attack harder.
Simple example
Picture a famous restaurant with one secret sauce. A competitor can’t get into the kitchen, so they do the next best thing: they order the dish hundreds of times, taste carefully, and reverse-engineer the recipe from the outside. They never see how it’s actually made. They just taste the output enough times to reproduce it. Their version isn’t perfect, but it’s close, it’s a lot cheaper, and the original restaurant did most of the work. A distillation attack is that, done to a model, one order at a time.