--- title: "Gary Stafford (@garystafford) | Hashnode" description: "Practical AI engineering: models, agents, and evaluations: from experiments to real-world applications. Read the latest articles by Gary Stafford on Hashnode." url: "https://hashnode.com/@garystafford" --- @garystafford # Gary Stafford @garystafford · New York City, USA · Joined October 2026 Practical AI engineering: models, agents, and evaluations: from experiments to real-world applications. ## Comments [Thanks, Julian. GPT-6 Sol at low reasoning got all 27 tasks right on the first pass, then scored 26/27 and 25/27 on the same tasks. On the regional failover task, the two later runs proposed plans costing $290 and $350 against a $260 optimum. The first pass would have put Sol/low on the shortlist. The repeats showed it missed our per-task requirement even though its overall accuracy stayed above 95%.](https://hashnode.com/posts/choosing-the-right-openai-model-and-reasoning-level/6abf0d0682c3fdfa1bc9ab22/comment/6ac374c1f30dd6567cecec37#reply-6af36a3bb23a0def963c9da0) Reply · Article · 1d ago · [Choosing the Right OpenAI Model and Reasoning Level](https://hashnode.com/posts/choosing-the-right-openai-model-and-reasoning-level/6abf0d0682c3fdfa1bc9ab22) [Thanks, Rishita. We used 27 saved responses for the calibration pilot. The judge agreed with 26 of the 27 original labels. Those examples came from development runs and the labels were reviewed with Codex, so I treated that agreement as evidence about how the rubric behaved on that set. For a decision that needs more confidence, I’d freeze the judge and rubric, then check them on fresh examples labeled by domain reviewers. I’d look especially closely at false passes on consequential mistakes, like a coordinator dropping a caveat that should have changed the plan.](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac5cd1c75fe3e50fe00c0ba#reply-6af3690098414658a8198689) Reply · Article · 1d ago · [How to Set Up Evals for a Complex Multi-Agent System](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03) [Thanks, Kartik. We checked both. In the Mars replay, each Director configuration had access to the same recorded specialist reports. The deterministic checks verified that the Director proposed an allowed set of actions and submitted it for approval. The semantic rubric separately checked whether the Director distinguished projections from observations and disclosed material shortfalls. For a dedicated handoff test, I’d make the caveat explicit in the fixture. Say a specialist reports normal air-processing throughput but says cabin air quality is unmeasured. I’d check that the Director carries that uncertainty into the plan or resolves it with new evidence. A plan can stay within its authority and still overstate what the team knows.](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac6966a95ff0ef484394708#reply-6af368f36f6fdf3b91b9a6b1) Reply · Article · 1d ago · [How to Set Up Evals for a Complex Multi-Agent System](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03) [Thanks, Kartik. Both versions passed all 12 evaluation runs, and the revised instructions were roughly twice as long. The agent received them on every model request. Even with tool calls dropping from 45 to 39, total input rose from 28,833 to 31,986 tokens, with zero cached input in either version. The extra input cost outweighed the savings from generating fewer output tokens, so estimated cost per successful run rose from $0.00606 to $0.00646, about 6.7%. The usage figures cover the whole request, including the instructions and the rest of the context, so the 6.7% is the measured change in run cost. The logs don’t break out how much of the extra input came from the instructions alone.](https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e/comment/6aeb3f6f3be35414dd042256#reply-6af3686198414658a8198230) Reply · Article · 1d ago · [From Hypothesis to Understanding: A Practical Framework for Applied AI Engineering Experiments](https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e) [Thanks, Abraam. The first rewrite mostly added procedural detail. It clarified when to stop, what to return, and how to handle retries. The key change was what counted as proof of a refund. If the refund tool explicitly confirmed success and returned a refund ID, the agent could use that response as evidence and skip the extra status lookup. After a timeout, it still had to check status before retrying. I did save a plain diff alongside the run results. In this case, the diff shows what the agent was allowed to skip, and the traces show what it actually skipped. All six fewer tool calls were refund-status lookups.](https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e/comment/6aec873f520975999b9d6a66#reply-6af3683898414658a81980bf) Reply · Article · 1d ago · [From Hypothesis to Understanding: A Practical Framework for Applied AI Engineering Experiments](https://hashnode.com/posts/from-hypothesis-to-understanding-a-practical-framework-for-applied-ai-engineering-experiments/6ae9ef828b8f5ca5929abb3e) [Great points. That’s a fair point. These results show how the models performed under the tested conditions; they don’t establish how they’ll perform in a particular production environment. Hypothetical, complex tasks let us control the evidence and verify answers, but I agree they don’t capture the full variability or domain specificity of real enterprise workloads. Real-world data, longer conversations, tool use, and noisy inputs could change both the model rankings and the cost and latency differences. A larger, more diverse evaluation would help determine which findings hold up. Again, this is designed as a fairly simple demonstration that introduces core concepts. The main takeaway is the evaluation method: define your acceptance criteria, test representative workloads, and compare the configurations that meet those requirements. These results are a starting point for that process, not a substitute for production-specific validation.](https://hashnode.com/posts/choosing-the-right-openai-model-and-reasoning-level/6abf0d0682c3fdfa1bc9ab22/comment/6abf56862c497b65968982f3#reply-6abf9cb1411d8e13ac31f688) Reply · Article · Oct 2 · [Choosing the Right OpenAI Model and Reasoning Level](https://hashnode.com/posts/choosing-the-right-openai-model-and-reasoning-level/6abf0d0682c3fdfa1bc9ab22) [I agree with your observations, especially on the deployment gap. Many enterprise use cases can be served well by previous-generation models. The question is whether a newer model delivers a measurable improvement that justifies adopting it. This demonstration uses one newer model family and a specific mix of hypothetical complex tasks, but the evaluation approach applies across use cases, domains, and model generations. The goal is to choose what meets your requirements, not automatically choose what’s newest. Adoption also varies widely, both across organizations and within them. In my experience, platforms like X and LinkedIn tend to amplify the cutting edge; enterprise teams are often focused on integration, governance, reliability, and proving business value. Those challenges don’t disappear with a better model.](https://hashnode.com/posts/choosing-the-right-openai-model-and-reasoning-level/6abf0d0682c3fdfa1bc9ab22/comment/6abf8fc7630bbc28eddd69c1#reply-6abf9c068d61b76f1d4c4c6c) Reply · Article · Oct 2 · 1 · [Choosing the Right OpenAI Model and Reasoning Level](https://hashnode.com/posts/choosing-the-right-openai-model-and-reasoning-level/6abf0d0682c3fdfa1bc9ab22)