FeedDiscussion
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The framing that "it didn't crash" tells you nothing about whether retrieval was relevant is the exact gap most RAG builds ignore until answers start drifting. Precision and recall at K get the attention, but MRR and NDCG are usually what expose a reranker that looks fine on averages while burying the best chunk at position four. Which of these do you actually gate on in practice, or do you track them for diagnosis and let end-to-end answer quality be the real bar?
A golden set tied to chunk IDs becomes stale as soon as the parser or chunker changes, even if the underlying evidence is unchanged. I prefer labeling source spans or evidence claims, then mapping those labels onto each index version before scoring. It is also useful to separate topical relevance from answer support: a chunk can be about the right subject yet lack the exact fact needed for generation. Reporting metrics by query slice and with hard negatives makes improvements much harder to hide inside one aggregate NDCG number.