Measuring retrieval and inference separately is exactly the discipline that turns a RAG demo into something you can debug, so I am glad you scoped it that narrowly on purpose. The failure mode I would watch on text-layer PDFs is tables and multi-column layouts, where extraction order scrambles the chunk before retrieval ever runs. Is your reproducible eval set catching extraction errors, or does it assume clean text going in?