yeah, the empty folder one is brutal because everything upstream looks fine. that's the whole problem with checking the summary.

for non-code work i score the end state, not what the agent says it did. "updated the docs" really means a file should be different, so the check goes and reads the file: did the target section change in the diff, does it actually contain the thing, any leftover TODO, does it still render. none of that needs a model.

the part that bites is writing the task so it has an end state you can look at. "update the docs" doesn't. "the auth section should describe the new token flow" does, you just go read that section. when i can't phrase it that way i take it as the same signal you did when you killed the handoff: no observable state proving it happened means the step is defined wrong, not just unchecked.