I thought retrieval in RAG was just one step. Turns out it's actually five or six different decisions hiding behind one word.
I'd been treating "retrieve the relevant chunks" as something simple and obvious — find the closest match, done. That's basically how I explained RAG to myself when I first got the hang of it: you've got some documents, you pull out the useful parts, hand them to an LLM, and get an answer back. Simple enough, or so I thought.
Then I actually sat down and looked into what retrievers are doing, and realized I'd been skipping over a step that matters a lot more than I gave it credit for.
So what is a retriever, really?
The simplest way I can put it now: a retriever takes a question and hands you back the documents that might actually help answer it. That's the whole job.
Say I've got a pile of documents about a company and someone asks, "How many days do employees get to work remotely?" I don't want to dump every document into the LLM and hope for the best. I want the system to go find the two or three pieces that are actually relevant to that question. That's what a retriever is for.
In my YouTube RAG project, it's the same idea — just with transcript chunks instead of company docs. Question comes in, retriever goes and searches, relevant chunks come out, LLM turns those into an answer.
Once that clicked, most of the rest of RAG made a lot more sense to me. But then I ran into something I genuinely hadn't considered: there isn't just one way to retrieve things. There are several, and they're not interchangeable.
The default: similarity search
The most basic approach is semantic similarity, and it's probably what most people picture when they think "RAG retrieval." You've already got embeddings for your documents. When someone asks a question, you embed that too, and then go looking for the document chunks whose embeddings sit closest to it.
So if I ask "how does the application store user information," a decent similarity search might surface a chunk that says something like "user data is persisted in PostgreSQL" — even though I never used the word "PostgreSQL" or "persisted" in my question. That's the whole appeal of doing this by meaning instead of keyword matching.
It works well. But there's one problem it doesn't solve: what happens when the top few results are all basically saying the same thing?
MMR: stop giving me the same answer five times
That's the exact problem MMR — Maximum Marginal Relevance — tries to fix. The idea is simple: give me relevant results, sure, but don't hand me five versions of the same thing.
Say I ask "what are the benefits of using RAG," and the retriever comes back with something like: RAG reduces hallucinations, RAG can reduce hallucinations, RAG helps prevent hallucinations, RAG reduces incorrect answers. Technically every one of those is relevant. Practically, I've learned exactly one thing four times.
MMR balances two competing questions for every candidate chunk — how relevant is this to the query, and how different is it from what I've already picked. So instead of four versions of "fewer hallucinations," I might end up with chunks that actually cover different ground: hallucination reduction, private data handling, freshness of information, citations. That's a much more useful set of four things to hand an LLM.
What if the question itself is the problem?
This is where multi-query retrieval got interesting for me. Take a question like "how does the system handle memory." Vague, right? Memory could mean conversation history, database storage, caching, something to do with the model itself — half a dozen reasonable interpretations, and the retriever has no way of knowing which one you meant.
Instead of betting everything on that one, ambiguous phrasing, a multi-query retriever uses an LLM to generate a few different versions of the question — "how is conversation history stored," "how does the system remember previous messages," "where is user conversation state maintained" — searches for all of them, and merges the results.
It's basically not putting all your eggs in one basket. Don't commit to one interpretation of an unclear question when you could just ask it a few different ways and combine what comes back. Once I saw it laid out like that, it made a lot of sense — but I never would've thought of it on my own.
Contextual compression took me a bit longer to get
This one didn't click right away. The setup: the retriever finds a document that's genuinely relevant. Good. But say that document is 2,000 words long, and only three sentences in it actually answer the question. Do you really want to ship all 2,000 words into your prompt?
Almost certainly not. A contextual compression retriever tries to keep the part that matters and discard the rest — so instead of "found a relevant doc, send the whole thing," you get "found a relevant doc, pull out what's useful, send that." Smaller, cleaner context, and presumably a better answer for it, since the model doesn't have to dig through 1,997 unnecessary words to find the three that count.
Why not just use the fanciest retriever every time?
That was my next question, honestly. If MMR and multi-query and compression all improve on plain similarity search, why wouldn't you just stack all of them and call it a day?
Because none of this is free. More steps means more waiting time, more API calls, more things that can break, more cost. And a lot of the time, plain similarity search is genuinely fine — if your documents are clean and your questions are reasonably direct, adding five extra layers of retrieval logic probably doesn't get you much.
So it turns out the real question isn't "what's the best retriever." It's "what's the right retrieval strategy for this problem" — which is a much less satisfying answer than I wanted, but a more honest one.
This changed how I think about RAG in general
When I first started learning this stuff, almost all of my attention went to the LLM side — which model to use, how to prompt it, how to squeeze a better answer out of it. Retrieval felt like a formality along the way.
Looking into retrievers flipped that for me. A lot of what determines whether a RAG answer is actually good gets decided before the LLM sees anything at all. If the retriever hands the model a weak or noisy set of chunks, no amount of clever prompting is going to save that answer — the model's just working with bad material. You can have the best prompt on earth, and it won't matter if the information behind it isn't what the model actually needed.
That's probably the biggest shift in how I think about RAG now: it's not really an LLM problem with a bit of search attached. It's closer to the other way around.
Still more to dig into here
The retrievers I've covered here are really just the starting point. There's also hybrid search, BM25, reranking, query rewriting, reciprocal rank fusion, and actually testing whether any of this is working — all things I haven't touched yet.
Reranking is what I'm most curious about next. Pulling back a big batch of maybe-useful chunks and then having a separate model figure out which ones are genuinely worth keeping feels like an important piece I've been skipping past entirely. That's where I'm headed next.