---
title: "Evelyn Wyatt (@Evelynwyatt) | Hashnode"
description: "Building AI Systems That Actually Work in Production. Read the latest articles by Evelyn Wyatt on Hashnode."
url: "https://hashnode.com/@Evelynwyatt"
---

@Evelynwyatt

# Evelyn Wyatt

@Evelynwyatt · Morresville · Joined August 2024

Building AI Systems That Actually Work in Production

[Website](https://www.itpathsolutions.com/)

## Comments

[This is an important distinction when moving AI agents from demos into production. Reporting that a button was clicked or an API call was triggered doesn't prove that the intended outcome was achieved. The system needs to verify the resulting state, whether that's a record being created, data being updated, or a workflow being completed. I think outcome verification should also include handling partial failures and duplicate actions, so an agent doesn't report success when only part of the task was completed. Reliable agents should be judged by verified results, not attempted operations.](https://hashnode.com/posts/an-ai-agent-should-report-outcomes-not-button-clicks/6aebf216dd442329dbe7f275/comment/6aec4a38b30513f7951d0300)

Comment · Article · Nov 5 · [An AI agent should report outcomes, not button clicks](https://hashnode.com/posts/an-ai-agent-should-report-outcomes-not-button-clicks/6aebf216dd442329dbe7f275)

[I think the distinction between models, workflows, and harnesses is important when building production-ready AI systems. A capable model alone doesn't guarantee reliable execution. The surrounding system needs to manage context, tool permissions, state, error recovery, and verification. One challenge is keeping workflows flexible enough to adapt to unexpected situations without giving agents unrestricted control. A well-designed harness should provide that balance while making execution observable and outcomes verifiable.](https://hashnode.com/posts/ai-agents-models-workflows-harnesses/6aec15495705da2d5d935dd6/comment/6aec47df66c6b5a6b15a3575)

Comment · Article · Nov 5 · [AI Agents Explained: Models, Workflows, and Harnesses](https://hashnode.com/posts/ai-agents-models-workflows-harnesses/6aec15495705da2d5d935dd6)

[One thing worth emphasizing with ServiceNow AI agents is that a successful response doesn't always mean the workflow completed correctly. At IT Path Solutions, we see the importance of verifying the actual record changes, approval states, and downstream actions rather than relying only on the agent's final message. A useful safeguard is to define the expected postconditions for each action and check them before reporting success. This makes failures easier to detect and prevents an agent from presenting an incomplete workflow as a completed task.](https://hashnode.com/posts/servicenow-ai-agent-mistakes/6ad09136b2ae051939d9ed94/comment/6ad0bd1f53e1c1382bb5f461)

Comment · Article · Oct 15 · [Common ServiceNow AI Agent Mistakes (and How to Avoid Them)](https://hashnode.com/posts/servicenow-ai-agent-mistakes/6ad09136b2ae051939d9ed94)

[A shared action log would make the setup much easier to audit, especially when several specialists are involved. I’d take it one step further and attach the verification evidence to each action as well—not just which agent ran the command, but what observable state changed afterward. That would help distinguish “the command executed” from “the requested outcome actually happened.” It could also make disagreements between agents much easier to resolve because the system has a common evidence trail rather than relying on each agent’s interpretation of the result.](https://hashnode.com/posts/a-crew-of-ai-agents-runs-my-homelab/6ac2ceba38aa7c69feb13799/comment/6ac38e1b940ed495a671a6c0)

Comment · Article · Oct 5 · [A Crew of AI Agents Runs My Homelab](https://hashnode.com/posts/a-crew-of-ai-agents-runs-my-homelab/6ac2ceba38aa7c69feb13799)

[The schema-versus-value distinction is useful, but there’s an interesting gap between “valid output” and “correct output.” A response can satisfy the schema perfectly while extracting the wrong invoice total, misclassifying a document, or producing a summary that misses a critical fact. I’d treat those as separate testing layers: deterministic tests verify the contract and how the application handles the response, while a smaller set of evaluation cases checks semantic correctness against known expectations. That also makes flaky tests easier to avoid without weakening coverage. The goal isn't to make every assertion deterministic; it's to make each test responsible for a property that can actually be judged reliably.](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38d069811b96acdb1f7dd)

Comment · Article · Oct 5 · [Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda)

[The distinction between finding relevant evidence and preserving sufficient evidence for reasoning is an important one here. A retrieval system can return highly relevant chunks and still fail if the answer depends on a definition, exception, or relationship that wasn't included in the final evidence bundle. I’d be interested in evaluating “evidence completeness” alongside retrieval precision and recall: not just whether the expected passage was retrieved, but whether all dependencies required to justify the answer survived selection and packing. That metric could also make the RAG-versus-long-context decision more concrete. If a task repeatedly requires broad evidence relationships, long context or structured parent expansion may be the better baseline even when top-k retrieval looks strong.](https://hashnode.com/posts/rag-vs-long-context-in-net/6ac12ef77048726fab66cd50/comment/6ac38c67c6334ea27e3bd211)

Comment · Article · Oct 5 · [RAG vs Long Context in .NET](https://hashnode.com/posts/rag-vs-long-context-in-net/6ac12ef77048726fab66cd50)

[The separation between core functionality and optional generation is the strongest part of this redesign. At IT Path Solutions, we’ve found that removing an LLM from the critical path is only half the reliability problem; the fallback still needs a clear product contract. If curated and generated content differ in difficulty, ambiguity, or expected behavior, the system can remain technically available while becoming inconsistent for users. Defining what the fallback must guarantee makes the architecture much stronger: the background model can improve the experience when it succeeds, but it should never change the guarantees of the core interaction. That turns AI into a true enrichment layer rather than a hidden dependency.](https://hashnode.com/posts/redesigning-an-llm-feature-for-reliability/6abaddbf7b74fb3693c520a7/comment/6abbb1aecbd61aa17ea053ab)

Comment · Article · Sep 29 · [When Personalisation Costs 15 Seconds: Redesigning an LLM Feature for Reliability](https://hashnode.com/posts/redesigning-an-llm-feature-for-reliability/6abaddbf7b74fb3693c520a7)

[The invalidation layer may be just as important as the persistence layer here. At IT Path Solutions, we’ve found that persistent agent state becomes much safer when the system can distinguish between information that was never verified and information that was verified but is no longer valid. In a changing environment, allowing an old verified state to remain trusted can be more dangerous than having no memory at all. I like the idea of tracking instance changes because it gives the system a concrete trigger for revalidation. It could be even more useful if the invalidation reason became part of the state, whether the target changed, credentials expired, a process disappeared, or the original evidence became stale. That would give the agent a much clearer basis for deciding when it can safely resume and when it needs to verify again.](https://hashnode.com/posts/remn-persistent-state-for-ai-agents-in-offensive-security/6ab4583908ea84028cc0e3ca/comment/6ab677cbd6c15bc911c8487b)

Comment · Article · Sep 25 · [REMN: Persistent State for AI Agents in Offensive Security](https://hashnode.com/posts/remn-persistent-state-for-ai-agents-in-offensive-security/6ab4583908ea84028cc0e3ca)

[The task-topology point highlights a subtle failure mode: physical isolation can make logically dependent tasks look independent. Two agents may have completely separate worktrees and still be coupled through a shared API contract, schema, dependency version, or architectural assumption. That suggests dependency detection should happen before execution, not only when branches reach the merge queue. Even a lightweight contract map could help the dispatcher identify likely coupling and decide whether tasks should run concurrently or against an explicit predecessor. Isolation prevents accidental filesystem interference, but it doesn't remove semantic dependencies. The more agents run in parallel, the more important that distinction becomes.](https://hashnode.com/posts/parallel-coding-agents-isolation/6ab4dc91cfcc71d3b3162208/comment/6ab6760e712075209877c4a2)

Comment · Article · Sep 25 · [One Agent, One Worktree: Isolation and Concurrency for Coding Agents](https://hashnode.com/posts/parallel-coding-agents-isolation/6ab4dc91cfcc71d3b3162208)

[The distinction also makes me think about how information should move between the two layers. Not everything an agent learns during a session deserves to become project memory. A temporary workaround, one-off debugging observation, and an architectural decision have very different lifetimes. A useful pattern could be to treat promotion into project memory as an explicit step: the agent identifies a finding, captures the evidence and scope, and only then proposes it as durable project knowledge. That would reduce the risk of turning every session observation into repository noise while still making important reasoning available to future sessions and teammates.](https://hashnode.com/posts/session-memory-is-not-project-memory-it-fixes-the-same-complaint/6ab3ae06ab5703ad22d98586/comment/6ab4f87b358512f172e0099c)

Comment · Article · Sep 24 · [Session memory is not project memory. It fixes the same complaint.](https://hashnode.com/posts/session-memory-is-not-project-memory-it-fixes-the-same-complaint/6ab3ae06ab5703ad22d98586)
