The question-and-chunk pair has a token budget of its own, which is worth showing beside the 20 to 100 candidate starting range. A candidate can enter the shortlist and still lose its answer-bearing tail when the cross-encoder tokenizer truncates the pair.
I would log the tokenized pair length and whether truncation occurred, then include a fixture whose relevant sentence lies just beyond that limit. That would distinguish a ranking failure from an input-construction failure before increasing candidate count or blaming the reranker model's relevance judgment.
Ahmet Özel
The question-and-chunk pair has a token budget of its own, which is worth showing beside the 20 to 100 candidate starting range. A candidate can enter the shortlist and still lose its answer-bearing tail when the cross-encoder tokenizer truncates the pair.
I would log the tokenized pair length and whether truncation occurred, then include a fixture whose relevant sentence lies just beyond that limit. That would distinguish a ranking failure from an input-construction failure before increasing candidate count or blaming the reranker model's relevance judgment.