The keyword-only fallback to a note's first piece can lose a correct candidate even after retrieval succeeded. If the matching term and answer sit near the end of a long note, the reranker sees its unrelated introduction and can discard the very note that keyword search found.

I would preserve the keyword match location and map it to the corresponding piece before reranking. One fixture with the answer only in the final piece would distinguish this from the separate top-20 candidate problem you described, and stage-level results would show whether the note disappeared before or during reranking.