The point that a reranker can't recover a document cut at position 51 is the one I'd test directly: log the pre-rerank rank of whatever the reranker ends up putting first, and if a meaningful share come from deep in the candidate list, widen the window before changing the model. It's also worth checking recall@50 of the fused list on its own, separate from final ranking quality, since that number caps everything after it. For identifier-style queries like calculate_tax_v2, BM25's default tokenizer may split on underscores or digits, so a code-aware analyzer (or an extra n-gram field) can matter more than the fusion weights. Since RRF ignores scores, have you considered weighting the keyword list more heavily for queries that look like identifiers or error codes, and measuring whether that beats plain k=60?