Your point that a generation problem downstream of a retrieval miss looks identical to a bad prompt is exactly what wasted a week for me once. I now split the eval so a low recall@k fails loud before I ever touch the prompt, otherwise I kept tuning the generator for a chunk it never received. I wrote about that split here: kartiknvjk.hashnode.dev/how-i-stopped-my-rag-ci-g… . How do you keep your expected_chunk_ids fresh when the chunking strategy changes?