Framing it as replay fidelity versus task continuation is sharp, because those two properties fail evals differently: replay tells you what the model saw, but not that the resumed task reached the same outcome. I have started asserting on post-recovery end state, not just transcript validity, since a valid transcript can still resume into a wrong branch. How do you test that a recovered approval or compaction state resumes to the same result the uninterrupted run would have produced?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Framing it as replay fidelity versus task continuation is sharp, because those two properties fail evals differently: replay tells you what the model saw, but not that the resumed task reached the same outcome. I have started asserting on post-recovery end state, not just transcript validity, since a valid transcript can still resume into a wrong branch. How do you test that a recovered approval or compaction state resumes to the same result the uninterrupted run would have produced?