Ask an LLM about your company's refund policy, and it will confidently make one up. Not because it's broken - because it genuinely doesn't know. It was trained on a snapshot of the public internet, not your internal docs, and it has no way to look anything up unless you give it one. RAG is how you give it one.
The short version
Retrieval-Augmented Generation means exactly what it says: before the model generates an answer, you retrieve relevant information and hand it that information as context. Instead of asking "what do you know about our refund policy," you first fetch the actual refund policy document, then ask "given this document, what's our refund policy" - and the model answers from text that's actually in front of it, not from memory.
Same question, two different starting points: what the model memorized, versus what you just handed it.
Why not just fine-tune the model instead?
Fine-tuning bakes new knowledge into the model's weights through additional training - it's a real technique, but it solves a different problem than RAG does. Fine-tuning is good at teaching a model a new style or skill - respond in this tone, follow this output format, handle this kind of task. It's a poor fit for teaching a model facts, especially facts that change: retrain the model every time a policy updates, and you're re-running an expensive training job for a one-line edit. RAG sidesteps that entirely - update the source document, and the next query automatically retrieves the new version. No retraining, no redeployment.
Why not just paste everything into the context window?
Modern context windows are large, and "just paste the whole knowledge base in" sounds appealing until you actually try it. Even where it fits, stuffing hundreds of pages into every single request is slow, expensive (you're paying per token, every time, for content the question may not even touch), and it degrades answer quality - models are measurably worse at using information buried in the middle of a long context than information that's front and center. Retrieval solves this by doing the filtering first: instead of "here's everything, figure it out," it's "here are the 3-5 passages that actually matter for this question."
Where RAG actually falls short
RAG is not magic, and it's worth knowing where it struggles before you build around it. It's only as good as retrieval: if the relevant chunk never gets found, the model never sees it, and it'll either say "I don't know" (the good outcome) or guess (the bad one). It's weak at questions that require synthesizing information scattered across many documents, or reasoning over an entire corpus at once - "summarize every complaint we got last quarter" is a harder fit than "what does the contract say about termination." And it adds real infrastructure: a vector database, an embedding pipeline, and a retrieval step that all need to be built, maintained, and monitored - covered later in this series.
What "good" RAG actually looks like
A well-built RAG system feels almost boring in the best way: ask it something the documents actually cover, and it answers accurately with a source you can check. Ask it something the documents don't cover, and it says so instead of guessing. That second behavior - knowing when to say "I don't know" - is often the real differentiator between a RAG system people trust and one they quietly stop using.