---
title: "Comment by Kartik N V J K on \"Memory Without Measurement Is Guesswork\" | Hashnode"
description: "The finding that adding memory scored equal or worse is the useful kind of negative result, because it exposes that the cases were passing on general agent qual"
url: "https://hashnode.com/posts/memory-without-measurement-is-guesswork/6a5606a3c563d2206a695133/comment/6a5664b04305465ab3bc660f"
---

[Discussion](https://hashnode.com/posts/memory-without-measurement-is-guesswork/6a5606a3c563d2206a695133)

Comment [Memory Without Measurement Is Guesswork](https://hashnode.com/posts/memory-without-measurement-is-guesswork/6a5606a3c563d2206a695133)

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Jul 14](https://hashnode.com/posts/memory-without-measurement-is-guesswork/6a5606a3c563d2206a695133/comment/6a5664b04305465ab3bc660f)

The finding that adding memory scored equal or worse is the useful kind of negative result, because it exposes that the cases were passing on general agent quality rather than recall. Locking a held-out split so prompt iteration cannot leak into final assessment is the discipline most eval harnesses drop first. Did judge variance stay small enough across the five providers that the deltas were trustworthy, or did you need multiple judge runs to separate signal from noise?

[António Hernández](https://hashnode.com/@fidelke)

[Jul 14](https://hashnode.com/posts/memory-without-measurement-is-guesswork/6a5606a3c563d2206a695133/comment/6a5664b04305465ab3bc660f#reply-6a566c99b105f0cad40ca507)

Yeah, the negative result was the useful part - every provider scoring equal or worse rules out bad implementations and points straight at the eval design.

Judge variance hit before the provider comparison. 20-point swings on identical cases, all traced back to no key\_decision anchor - the judge was scoring general quality instead of recall. Pinned the anchor first, then the deltas were trustworthy.

Have you run into something similar - measuring the wrong layer and only realizing it after the numbers looked stable?
