Discussion on "Evals: How Do You Actually Know Your AI Feature Works?"

Nagappan S · 2026-10-06T05:48:33.829Z

You ship a new AI feature. It works great in your testing — you type in a few prompts, the answers look right, you demo it to your team, everyone nods. Three weeks later, a user reports that the assis

The pitfall you flag, eval scores climbing while user complaints stay flat, is the one that bit me hardest. A judge-model update changed its output format, my parser quietly fell back to a middle score, and the dashboard drifted up while nothing had improved. I wrote about this here: kartiknvjk.hashnode.dev/i-built-an-llm-eval-frame… Do you re-validate your judge against human labels on a set cadence, or only when a number looks wrong?

That silent fallback is a nasty one. A parse failure that returns a middle score looks like a result. I'd make the parser fail loudly and track the parse-failure rate as its own metric.

For the judge, I'd do both. Re-check it against a small human-labelled set on a fixed cadence, and also whenever the judge model or its prompt changes. Pinning the judge version helps, because then drift only happens when you choose it. A sudden shift in the score distribution with no system change is a good trigger too.

The exchange in the responses is the most useful part for me: scoring the path, not just the final answer. A clean demo prompt tells you the model can answer. It says nothing about whether the agent backs off on a 429, stops instead of looping, or carries a changed instruction forward.

When we scope AI features for clients, we now treat the eval set as part of the deliverable and agree on it before building. Twenty or thirty cases, each with a written expected behaviour, including the awkward ones: the user changes the request halfway, a tool returns garbage, the right answer is "I cannot do that". The feature is done when those pass, and the same set runs on every model swap or prompt change.

It also settles arguments that otherwise drag on. "The assistant feels worse this week" becomes "these four cases regressed after Tuesday's prompt change", which is something a team can fix. The hardest part is usually getting the business owner to write the expected answers, because that is where the real definition of the feature lives.

Growing the eval set after every user-reported bug is useful, but I would version the dataset and keep the previous release’s cases as a stable comparison slice. Otherwise a score can fall because coverage improved, or rise because difficult cases disappeared, without the underlying system changing.

For trajectory evaluation, acceptable alternatives should be explicit too. Two agents may complete the same task using different valid tool sequences, so comparing against one canonical path can penalize a better strategy. Constraints on required effects, forbidden actions and resource use are a more durable contract than exact step-for-step matching.

Agreed on both. Versioning the dataset and keeping the previous release's cases as a frozen slice is what separates "the system got worse" from "the coverage got better". I'd report the stable slice and the full set side by side.

On trajectories, constraints are the better contract: required effects, forbidden actions and resource limits, rather than one canonical path. That way a better strategy isn't penalised just for taking a different route.

Reporting both sets also makes new-case regressions actionable: keep case identifiers stable and show which failures belong to the frozen slice versus the added coverage. Otherwise readers still have to infer whether the release broke old behavior or uncovered an older weakness.

For trajectory constraints, I would check the required effect in the resulting system state, not only the tool-call log. A plausible call sequence can still leave the task incomplete, while a shorter valid sequence may satisfy the same contract.

Demo prompts hide the failure mode. The eval that matters is a case where the user changes the ask mid task, or the tool returns a 429 and the agent has to decide whether to retry or stop. Do you score that path, or only the first clean answer? iin1006h11

You should score that path, and it's where most of the signal is. I'd add dedicated trajectory cases for exactly those scenarios: a mid-task change of request, a tool returning a 429 or timeout, a malformed tool response.

What gets scored there isn't the final answer alone. It's the decisions: did it back off and retry a bounded number of times, did it stop and report instead of looping, did it carry the new instruction forward, and did it avoid repeating side-effecting calls. Rule-based checks cover most of it (call counts, retry limits, no duplicate writes), with an LLM judge only for the fuzzier "did it handle the changed ask correctly" part.

Clean-path-only evals tend to look great right up until a flaky dependency shows up in production. How are you injecting failures in your setup, mocked tools or real chaos in staging?

Search Hashnode

Evals: How Do You Actually Know Your AI Feature Works?

Responses8