Building scalable products at the intersection of Full Stack Engineering & AI
I build scalable web apps, experiment with AI, and turn complex engineering problems into working products.
Markdown supported · ⌘↵ to post
The point that a reranker can't recover a document cut at position 51 is the one I'd test directly: log the pre-rerank rank of whatever the reranker ends up putting first, and if a meaningful share come from deep in the candidate list, widen the window before changing the model. It's also worth checking recall@50 of the fused list on its own, separate from final ranking quality, since that number caps everything after it. For identifier-style queries like calculate_tax_v2, BM25's default tokenizer may split on underscores or digits, so a code-aware analyzer (or an extra n-gram field) can matter more than the fusion weights. Since RRF ignores scores, have you considered weighting the keyword list more heavily for queries that look like identifiers or error codes, and measuring whether that beats plain k=60?
Ahmet Özel
The two-researcher analogy assumes both ranked lists identify the same kind of item. In practice one search leg may return document IDs while the other returns passage IDs, and RRF cannot recognize agreement between them unless the fusion key is defined consistently.
I would record the ranking unit at each stage and test a document returned by the keyword leg whose relevant passage is returned by the dense leg. Decide explicitly whether fusion happens at passage or document level, then preserve the mapping for the reranker. That makes a high rank in both lists a well-defined claim instead of an artifact of incompatible identifiers.