Discussion on "Evals: How Do You Actually Know Your AI Feature Works?"

Nagappan S · 2026-10-06T05:48:33.829Z

You ship a new AI feature. It works great in your testing — you type in a few prompts, the answers look right, you demo it to your team, everyone nods. Three weeks later, a user reports that the assis

Demo prompts hide the failure mode. The eval that matters is a case where the user changes the ask mid task, or the tool returns a 429 and the agent has to decide whether to retry or stop. Do you score that path, or only the first clean answer? iin1006h11

You should score that path, and it's where most of the signal is. I'd add dedicated trajectory cases for exactly those scenarios: a mid-task change of request, a tool returning a 429 or timeout, a malformed tool response.

What gets scored there isn't the final answer alone. It's the decisions: did it back off and retry a bounded number of times, did it stop and report instead of looping, did it carry the new instruction forward, and did it avoid repeating side-effecting calls. Rule-based checks cover most of it (call counts, retry limits, no duplicate writes), with an LLM judge only for the fuzzier "did it handle the changed ask correctly" part.

Clean-path-only evals tend to look great right up until a flaky dependency shows up in production. How are you injecting failures in your setup, mocked tools or real chaos in staging?

The exchange in the responses is the most useful part for me: scoring the path, not just the final answer. A clean demo prompt tells you the model can answer. It says nothing about whether the agent backs off on a 429, stops instead of looping, or carries a changed instruction forward.

When we scope AI features for clients, we now treat the eval set as part of the deliverable and agree on it before building. Twenty or thirty cases, each with a written expected behaviour, including the awkward ones: the user changes the request halfway, a tool returns garbage, the right answer is "I cannot do that". The feature is done when those pass, and the same set runs on every model swap or prompt change.

It also settles arguments that otherwise drag on. "The assistant feels worse this week" becomes "these four cases regressed after Tuesday's prompt change", which is something a team can fix. The hardest part is usually getting the business owner to write the expected answers, because that is where the real definition of the feature lives.

Growing the eval set after every user-reported bug is useful, but I would version the dataset and keep the previous release’s cases as a stable comparison slice. Otherwise a score can fall because coverage improved, or rise because difficult cases disappeared, without the underlying system changing.

For trajectory evaluation, acceptable alternatives should be explicit too. Two agents may complete the same task using different valid tool sequences, so comparing against one canonical path can penalize a better strategy. Constraints on required effects, forbidden actions and resource use are a more durable contract than exact step-for-step matching.

Search Hashnode

Evals: How Do You Actually Know Your AI Feature Works?

Responses(4)