Great point. I think that's one of the biggest challenges with persistent memory systems deciding what deserves to become memory versus what should remain transient context.
My view is that not everything should be stored verbatim. Stable facts (user preferences, long-term goals, project context, etc.) are good candidates for persistence, while temporary conversational details are often better summarized or discarded. Compression, however, introduces the risk of losing information that later becomes important, so retrieval quality becomes just as critical as storage.
That's also why I find work reuse interesting. Rather than replaying or compressing entire histories, systems can sometimes reuse previously computed work when the new request is semantically similar, avoiding unnecessary context reconstruction altogether. There isn't a single best approach, it's often a combination of retrieval, summarization, and selective reuse depending on the application.
Thanks for bringing this up!
The framing of an LLM call as a stateless pure function is the right mental model, and it makes clear that every "memory" feature is really just history reconstruction you pay for on every call. The trade-off I keep running into is that naive full-history replay balloons token cost and latency as a conversation grows, so at some point you need summarization or retrieval to bound the context. When you move to a memory architecture, how do you decide what to keep verbatim versus compress, and do you evaluate whether that compression is dropping facts the model later needs?