Retrieval Evaluation: Is Your RAG System Finding the Right Thing?

A RAG system can generate a great-sounding answer from completely wrong retrieved chunks. Evaluating retrieval separately from the final answer is how you catch that.

Two different failures, easy to conflate

A RAG pipeline can fail in two separate places: retrieval can fetch the wrong chunks, or generation can write a bad answer even from good chunks. If you only judge the final answer, you can't tell which broke - "the answer was wrong" doesn't tell you whether to fix chunking, embeddings, or the prompt.

Precision and recall, applied to retrieval

Precision asks: of the chunks retrieved, how many were actually relevant? Recall asks: of all chunks relevant somewhere in your corpus, how many did you retrieve? A retriever can have high precision but low recall - these pull in different directions and both matter.

Building a test question set

You need real questions paired with the specific chunks that should be retrieved for each - built by hand from your domain, not guessed generically. Without this ground truth, you're evaluating retrieval by vibes, which doesn't catch regressions when you change chunking or swap embedding models.

Typical failure sources

Chunking that splits a relevant fact across two chunks so neither alone looks relevant enough. An embedding model that's a poor fit for your domain's vocabulary. Missing synonyms - a user asks about "deleting an account" but your docs only say "closing an account." Each needs a different fix, which is why you need to know which is actually happening.

EXAMPLE

A tiny worked example with 3 chunks in the corpus and a query "How do I cancel my subscription?": Corpus chunks: [1] "Subscriptions renew automatically each month unless cancelled." [2] "To cancel, go to Settings > Billing > Cancel Plan." (the actually relevant chunk) [3] "Our refund policy covers the first 14 days." Retriever returns, ranked: [1, 3, 2] Precision@3 = 1/3 (only 1 of the 3 returned chunks is truly relevant to the specific question) Recall@3 = 1/1 = 100% (the relevant chunk WAS retrieved somewhere in the top 3) But: relevant chunk ranked LAST (position 3), which is exactly what context-precision-style metrics penalize - a generation step reading only the top 1-2 chunks would miss it entirely. Fix directions this points to: - If [2] should rank higher: check embedding similarity for that specific phrasing, or add a synonym like "cancel"/"end subscription" to the retrieval index. - If [1] and [3] shouldn't be retrieved at all: check chunk granularity - they may be too broadly similar to too many queries.

🛠️ EXERCISE — TRY IT YOURSELF

Build a tiny retrieval evaluation set for a RAG system (real or hypothetical) and score it by hand.

Pick 8-10 real questions your RAG system should handle, and identify by hand which specific chunk(s) should be retrieved for each.
Run each question through your retriever and record the top 3-5 chunks it actually returns.
For each question, mark which returned chunks are truly relevant vs. not.
Compute rough precision (relevant / returned) and recall (relevant found / relevant that exist) for a few questions.
Identify at least one failure and classify it as a chunking, embedding, or synonym/vocabulary issue.

✅ SELF-CHECK

☐ Could you clearly separate at least one retrieval failure from a generation failure using this exercise?
☐ For the failure you found, could you point to a specific likely cause (chunk boundary, embedding mismatch, missing synonym) rather than just 'it didn't work'?

QUICK QUIZ

A RAG system's retriever has high recall but low precision. What does that mean in practice?