---
title: "Comment by indiainfranotes on \"Evals: How Do You Actually Know Your AI Feature Works?\" | Hashnode"
description: "Demo prompts hide the failure mode. The eval that matters is a case where the user changes the ask mid task, or the tool returns a 429 and the agent has to deci"
url: "https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac4920a811880132ddab0ed"
---

[Discussion](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c)

Comment [Evals: How Do You Actually Know Your AI Feature Works?](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c)

[indiainfranotes](https://hashnode.com/@indiainfranotes)

[Oct 6](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac4920a811880132ddab0ed)

Demo prompts hide the failure mode. The eval that matters is a case where the user changes the ask mid task, or the tool returns a 429 and the agent has to decide whether to retry or stop. Do you score that path, or only the first clean answer? iin1006h11

[Nagappan S](https://hashnode.com/@imkarthik03)

[Oct 6](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac4920a811880132ddab0ed#reply-6ac49880b009d278956d5789)

You should score that path, and it's where most of the signal is. I'd add dedicated trajectory cases for exactly those scenarios: a mid-task change of request, a tool returning a 429 or timeout, a malformed tool response.

What gets scored there isn't the final answer alone. It's the decisions: did it back off and retry a bounded number of times, did it stop and report instead of looping, did it carry the new instruction forward, and did it avoid repeating side-effecting calls. Rule-based checks cover most of it (call counts, retry limits, no duplicate writes), with an LLM judge only for the fuzzier "did it handle the changed ask correctly" part.

Clean-path-only evals tend to look great right up until a flaky dependency shows up in production. How are you injecting failures in your setup, mocked tools or real chaos in staging?
