Yeah, the negative result was the useful part - every provider scoring equal or worse rules out bad implementations and points straight at the eval design.
Judge variance hit before the provider comparison. 20-point swings on identical cases, all traced back to no key_decision anchor - the judge was scoring general quality instead of recall. Pinned the anchor first, then the deltas were trustworthy.
Have you run into something similar - measuring the wrong layer and only realizing it after the numbers looked stable?
Kartik N V J K
The finding that adding memory scored equal or worse is the useful kind of negative result, because it exposes that the cases were passing on general agent quality rather than recall. Locking a held-out split so prompt iteration cannot leak into final assessment is the discipline most eval harnesses drop first. Did judge variance stay small enough across the five providers that the deltas were trustworthy, or did you need multiple judge runs to separate signal from noise?