How RAG Actually Works (It's Mostly Retrieval)
A plain-English guide to how RAG works. Retrieval-augmented generation is really a prompt trick, and the retriever is where your system succeeds or fails.
- RAG
- LLMs
- AI Engineering
Ask a raw language model about your company's refund policy and it does one of two things: admits it has no idea, or confidently invents one. Neither is useful. The model was never trained on your documents, so it has no way to know what's in them.
The usual instinct is "let's train the model on our data." Almost always, that's the wrong move, and it's the fastest way to misunderstand how RAG works. What you actually want is retrieval-augmented generation, and it solves the problem with a trick that sounds too simple to work.
In most production apps, you retrieve the right information first, then give the model that context before it answers.
How RAG works in practice: retrieval before generation
A large language model only knows what it saw during training. Anything newer, private, or specific to your organization is invisible to it.
RAG fixes this by editing the prompt before it ever reaches the model. A component called the retriever searches a knowledge base (really just a database of trusted documents), pulls the passages most relevant to the question, and pastes them into the prompt alongside the user's original query. The model reads all of it and answers.
That's the whole idea. From the user's side nothing looks different. They type a question, they get an answer, maybe with a little added latency. The real work happens in what got added to the prompt before the model saw it.
It works because language models are genuinely good at one thing: using context. Give a model relevant text and it will fold that text into its answer, even for facts that were nowhere in its training data.
Why not just fine-tune the model instead?
This is the question that comes right after the "train it on our docs" one. Fair question, clear answer: fine-tuning is usually the wrong first tool for keeping a model up to date with changing facts.
Fine-tuning mostly shapes how a model responds: its tone and format, the style of its answers. It has a much weaker grip on what the model actually knows. So if the goal is "make the model aware of this week's pricing," fine-tuning is slow, expensive, and unreliable at exactly the thing you wanted.
RAG sidesteps all of that. Your knowledge lives in a database, and you update it the way you'd update any other database. A new policy drops on Monday? Add the document, and every answer from that moment on can use it. No retraining, no GPU bill, no waiting.
That single property, knowledge you can edit in place, is why RAG won for most real applications.
The retriever is where RAG lives or dies
Here's the part people underestimate: the LLM is the easy half of a RAG system. Plugging in a model and getting fluent text back is a solved problem. Your quality comes from the retriever.
The retriever keeps an index of your documents so it can search them fast. When a question comes in, it scores every candidate document by how relevant it is (usually some measure of how similar the document's text is to the question) and returns the highest-scoring ones. Everything downstream depends on that ranking being good.
In practice, when a RAG system gives a wrong or vague answer, the model usually isn't the culprit. The retriever handed it the wrong passages, and no model can reason its way out of bad context. This is worth turning into a habit: when a RAG answer is bad, look at what got retrieved before you touch anything else. Most of the time the bug is right there.
What one RAG request looks like, step by step
Walk through a single question end to end and the whole thing stops feeling mysterious:
- The user submits a prompt.
- The system routes it to the retriever instead of straight to the model.
- The retriever queries the knowledge base and returns the most relevant documents.
- The system builds an augmented prompt: the retrieved text plus the original question.
- The model reads that combined prompt and generates the answer.
Five steps. Only one of them involves the model, and it's the step you'll spend the least time on.
What I check first when RAG fails
When a RAG answer is wrong, I don't start by changing the model. I inspect the retrieved chunks first:
- Did the right document appear in the top results?
- Was the chunk too small or too large?
- Did metadata filtering remove the correct source?
- Did the query need keyword search instead of only vector search?
- Did the prompt clearly tell the model to answer only from retrieved context?
Most RAG bugs are visible before the LLM ever generates a token.
Put your effort into retrieval
If you take one thing from this: RAG is not the model. The model is a commodity you can swap out in an afternoon. The system is your retriever, your documents, and whether the right passage lands in the prompt at the right moment.
Get that part right and a mid-tier model gives great answers. Get it wrong and the best model on the market still hallucinates, because you handed it the wrong context and asked it to be smart anyway.
If you're building one of these and the answers aren't landing, start with retrieval — that's where most of the signal is. If it'd help to have another set of eyes on it, I'm happy to talk.