Discussion on "RAG evaluation: how to know if your retrieval is actually good"

Amit Shukla · 2026-09-29T13:15:26.291Z

You built a RAG app, asked it five questions in the demo, and every answer sounded right. That is not evaluation, that is vibes. The moment a real user asks something slightly off the beaten path, ret

Your point that a generation problem downstream of a retrieval miss looks identical to a bad prompt is exactly what wasted a week for me once. I now split the eval so a low recall@k fails loud before I ever touch the prompt, otherwise I kept tuning the generator for a chunk it never received. I wrote about that split here: kartiknvjk.hashnode.dev/how-i-stopped-my-rag-ci-g… . How do you keep your expected_chunk_ids fresh when the chunking strategy changes?

Same week lost here, so I feel this one. I stopped storing chunk IDs for exactly that reason. Each eval case now stores the source doc plus the exact answer text, and a retrieved chunk counts as a hit if it contains that text. Change the chunking and the eval set still works, no relabeling. Nice write-up on the CI gate, thanks for sharing it.

Good rundown of the retrieval-vs-generation split, that's the part most RAG writeups blur together. One thing worth adding to "where teams get this wrong": the ground truth itself is only as stable as the chunking scheme it was labeled against. expected_chunk_ids pinned to one chunking strategy don't reliably still point at the same content after you change chunk boundaries, so a before/after recall@k comparison across a chunking change can be comparing against a stale definition of "the right chunk" rather than the new one, unless the ground truth gets re-derived by content match rather than trusted by ID. And on the LLM-judge faithfulness check, it's worth calibrating the judge itself before trusting it across the whole eval set — run it against a small hand-labeled set of known SUPPORTED/UNSUPPORTED cases first. A judge that's quietly biased toward SUPPORTED (a common failure mode) will launder unfaithful answers through a metric that looks like it's catching them.

Both fair, and the first one is exactly where I ended up. Labels now point at the answer text instead of chunk IDs, so a chunking change doesn't quietly move the goalposts. Agree on the judge too. A small hand-labeled set to check it against first is cheap, and a judge that leans SUPPORTED is worse than no judge, since it looks like it's working. Might fold both into the post.

Search Hashnode

RAG evaluation: how to know if your retrieval is actually good

Responses(4)