---
title: "Discussion on \"Memory Without Measurement Is Guesswork\" | Hashnode"
description: "Discussion on \"Memory Without Measurement Is Guesswork\". Five memory providers, one eval harness, and the discovery that adding memory context made every provider score equal or worse."
url: "https://hashnode.com/posts/memory-without-measurement-is-guesswork/6a5606a3c563d2206a695133"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Memory Without Measurement Is Guesswork](https://agentbuildlog.hashnode.dev/memory-without-measurement-is-guesswork?utm_source=hashnode&utm_medium=feed)

Five memory providers, one eval harness, and the discovery that adding memory context made every provider score equal or worse.

Jul 14 · 21 min read · agentbuildlog.hashnode.dev  
Read the article

-   [#ai-agents](https://hashnode.com/tag/ai-agents)
-   [#llm](https://hashnode.com/tag/llm)
-   [#python](https://hashnode.com/tag/python)
-   [#testing](https://hashnode.com/tag/testing)

## Written by

[António Hernández](https://hashnode.com/@fidelke)

@fidelke · Košice, Slovakia

  
[View profile](https://hashnode.com/@fidelke)

Senior Implementation Engineer at Atlan

I'm an engineer who builds systems and writes about what actually happens when I do. I'm currently working through the craft of AI agent engineering from the ground up - documenting the experiments, the failures, and the moments that change my mental model. No polished takes. Just the build log.

## Responses 2

Markdown supported · ⌘↵ to post

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Jul 14](https://hashnode.com/posts/memory-without-measurement-is-guesswork/6a5606a3c563d2206a695133/comment/6a5664b04305465ab3bc660f)

The finding that adding memory scored equal or worse is the useful kind of negative result, because it exposes that the cases were passing on general agent quality rather than recall. Locking a held-out split so prompt iteration cannot leak into final assessment is the discipline most eval harnesses drop first. Did judge variance stay small enough across the five providers that the deltas were trustworthy, or did you need multiple judge runs to separate signal from noise?

[António Hernández](https://hashnode.com/@fidelke)

[Jul 14](https://hashnode.com/posts/memory-without-measurement-is-guesswork/6a5606a3c563d2206a695133/comment/6a5664b04305465ab3bc660f#reply-6a566c99b105f0cad40ca507)

Yeah, the negative result was the useful part - every provider scoring equal or worse rules out bad implementations and points straight at the eval design.

Judge variance hit before the provider comparison. 20-point swings on identical cases, all traced back to no key\_decision anchor - the judge was scoring general quality instead of recall. Pinned the anchor first, then the deltas were trustworthy.

Have you run into something similar - measuring the wrong layer and only realizing it after the numbers looked stable?
