Semantic Search Isn't Enough: A Practitioner's Guide to Hybrid Retrieval

Agree - the reranker is the right layer for the abstain decision, but its score alone is a shaky thing to threshold on, for a specific reason.

Rerankers are trained mostly on "which of these results is the right one," and rarely on queries where the answer is "none of them." So the point worth catching, a query with no good answer, the model is least reliable: it still returns a confident score for the least-bad result. Converting that score into a probability and thresholding on it only sharpens what the model already learned, and it never really learned what "nothing fits" looks like.

Couple of cheaper signals tend to help more than the reranker score by itself:

First, whether the keywords actually matched. Rank fusion only looks at the order results came back in, not how strong each match was, so it discards BM25's signal that no query term really landed. That low-match signal is the tell for a no-answer query. Abstaining when neither the keyword match nor the reranker is strong catches the classic confident-wrong case: a weak keyword hit plus a vaguely related semantic result.

Second, a per-query-type threshold instead of a global one. Navigational, exploratory, and long-tail queries score completely differently, so a single bar over-blocks some intents and under-blocks others. Even a rough three-way split beats one number.

None of it is measurable unless the test set includes queries that should return nothing: the unanswerable, out-of-domain ones. Test sets built only from queries that have a good answer can't measure the abstain decision at all, which is why it usually goes untuned.