Making abstention the product and converging all three layers on a single INSUFFICIENT_EVIDENCE token is the part I like most, since a docs assistant that guesses is worse than one that says it does not know. Using a stronger judge model than the generator for faithfulness and context precision is the detail most people get backwards, and enforcing retrieval recall as a CI ratchet so a regression fails the build is exactly how you keep a RAG system honest over time. The post-hoc citation validation that refuses to cache ungrounded answers is a clean way to stop a bad answer from becoming a permanent one.
KN
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Thanks, glad it landed. I used a stronger judge, but even that can't really be trusted without evaluating it on its own. I see you've been working on RAG evals including LLM judges, so I'm curious how your experience has been with evaluating the judge itself.
Making abstention the product and converging all three layers on a single INSUFFICIENT_EVIDENCE token is the part I like most, since a docs assistant that guesses is worse than one that says it does not know. Using a stronger judge model than the generator for faithfulness and context precision is the detail most people get backwards, and enforcing retrieval recall as a CI ratchet so a regression fails the build is exactly how you keep a RAG system honest over time. The post-hoc citation validation that refuses to cache ungrounded answers is a clean way to stop a bad answer from becoming a permanent one.