The replay method is strongest when it separates model drift from retrieval drift: reusing the same retrieved context makes the comparison fair, but can miss a changed retrieval path. In my work on AiEnsured agentest, an Agentic AI Testing platform, an a17 multi-hop RAG run retrieved the documents containing the answer yet still replied, “Not found in the official documents.” That’s why I’d pair outcome agreement with citation and retrieval checks during disagreement review; your stratified sampling and reviewer-corrected cases offer a practical place to do it.