My first call to an LLM failed before a single chunk reached it. The model the course uses has been retired. Once that was fixed, I found a bigger problem: my RAG pipeline confidently answered a quest
Markdown supported · ⌘↵ to post
The gap I'd flag is that the threshold and the refusal line only catch the case where nothing relevant came back at all. They don't catch the harder one, where a chunk clears 0.3 on topic but the model still writes an answer that goes beyond what that chunk actually supports, or blends two chunks into a claim neither makes on its own. The "confidence" you're returning is really just the top retrieval score, not a measure of whether the generated sentence is actually entailed by the sources you're citing.
The fix I've used for that is a cheap post-hoc check rather than a stronger prompt: after generation, verify each cited source snippet against the answer text (even a simple substring/overlap or embedding-similarity check between claim and citation) and drop or flag any sentence whose citation doesn't back it up. It won't stop the model from trying to overreach, but it stops an overreach from reaching the user silently, which a prompt instruction alone can't guarantee once the threshold has already let the chunk through.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
Treating a confident answer with no supporting chunk as a bug rather than a quirk is exactly the right lesson to take from this one.
Worth knowing the shape it keeps as you scale: abstention is not something you reliably prompt your way into, because the model cannot distinguish "this context is thin" from "this context is enough". The version that held up for me is a threshold before generation rather than an instruction inside it. If the top retrieval score is below a floor you derived from a labelled set, you return the refusal yourself and never call the model at all. It also gives you a number you can tune, which "say I don't know" never does.