---
title: "Comment by Abraam Gotsiridze on \"From Hypothesis to Understanding: A Practical Framework for Applied AI Engineering Experiments\" | Hashnode"
description: "\"Compare against a baseline that already works well\" is the line I'd want more agent write-ups to follow. A lot of \"the model rewrote its own prompt and got bet"
url: "https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e/comment/6aec873f520975999b9d6a66"
---

[Discussion](https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e)

Comment [From Hypothesis to Understanding: A Practical Framework for Applied AI Engineering Experiments](https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e)

[Abraam Gotsiridze](https://hashnode.com/@abraamgotsiridze)

[6d ago](https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e/comment/6aec873f520975999b9d6a66)

"Compare against a baseline that already works well" is the line I'd want more agent write-ups to follow. A lot of "the model rewrote its own prompt and got better" results start from a prompt that never had a fair chance, so it's refreshing to read one where the baseline passed and the honest finding was about cost, not accuracy.

The step 6 example is the one I'd underline: fewer tool calls, but higher cost per successful task. Tool calls are easy to count in a trace, so they're the number people reach for, and here they pointed the wrong way. Reporting cost per success next to the cost of producing the revision keeps a "cheaper" version from hiding what it took to get there.

One small addition that could fit in step 5: save a plain diff of the baseline and the frozen replacement instructions next to the results. When a revision saves money, the diff usually shows why, for example a dropped check or a shorter tool list. That's what tells you whether the saving is likely to hold on a different workload.

Out of curiosity, did the model's replacement instructions mostly add rules or remove them?

[Gary Stafford](https://hashnode.com/@garystafford)

[1d ago](https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e/comment/6aec873f520975999b9d6a66#reply-6af3683898414658a81980bf)

Thanks, Abraam. The first rewrite mostly added procedural detail. It clarified when to stop, what to return, and how to handle retries. The key change was what counted as proof of a refund. If the refund tool explicitly confirmed success and returned a refund ID, the agent could use that response as evidence and skip the extra status lookup. After a timeout, it still had to check status before retrying.

I did save a plain diff alongside the run results. In this case, the diff shows what the agent was allowed to skip, and the traces show what it actually skipped. All six fewer tool calls were refund-status lookups.