☀️ AI Morning Minute: Chunking
Before an AI can answer questions about your documents, something has to slice them into pieces first. That slicing is where a lot of AI quietly goes wrong.
You’ve probably seen an AI tool that lets you “chat with your documents.” Upload a big PDF, ask questions, get answers. Feels like magic. But there’s an unglamorous step happening behind the curtain that decides whether those answers are any good, and almost nobody talks about it. It’s called chunking, and it’s basically about where you cut.
What it means
Chunking is the process of splitting a document into smaller pieces before an AI can search through it. A system like RAG (the setup that lets AI answer from your files) can’t just swallow a 200-page manual whole. So it breaks the document into chunks, maybe a few paragraphs each, turns each chunk into numbers it can search, and stores them. When you ask a question, it grabs the chunks that best match and feeds those to the model.
The whole trick is where you make the cuts. Slice mid-thought and you hand the AI half an idea.
Why it matters
Bad chunking makes the AI look dumb when the AI is fine. If a system splits a document clumsily, it pulls back fragments that are missing the context, and the model fills the gap with a guess. A lot of “the AI made things up” complaints trace back here. The model wasn’t broken. The slicing was.
The size of the pieces is a real tradeoff. Tiny chunks are precise but lose the surrounding context. Big chunks keep context but drag in noise and cost more to process. Most production systems land somewhere around 200 to 500 words per chunk, though honestly the “right” size depends a lot on your documents, and people argue about it constantly.
No fancy model saves you from it. This is the part teams learn the expensive way. You can plug in the best, priciest AI on the market, but if your chunking is sloppy, the model never sees the right information in the first place. Garbage pieces in, garbage answer out.
Simple example
Think about ripping a recipe out of a cookbook, but you tear the pages at random spots instead of at the recipe boundaries. Now the ingredient list is on one scrap and the instructions are on another, and the oven temperature got torn clean in half. When you go looking for “how long do I bake this,” you grab a scrap that just says “for 25.” Twenty-five what? At what temperature? You’re stuck.
Chunk a document the same careless way and the AI ends up holding those same useless scraps.