---
title: "Comment by Ahmet Özel on \"Evals: How Do You Actually Know Your AI Feature Works?\" | Hashnode"
description: "Growing the eval set after every user-reported bug is useful, but I would version the dataset and keep the previous release’s cases as a stable comparison slice"
url: "https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac53cedbe529719bcce1bb7"
---

[Discussion](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c)

Comment [Evals: How Do You Actually Know Your AI Feature Works?](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c)

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 6](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac53cedbe529719bcce1bb7)

Growing the eval set after every user-reported bug is useful, but I would version the dataset and keep the previous release’s cases as a stable comparison slice. Otherwise a score can fall because coverage improved, or rise because difficult cases disappeared, without the underlying system changing.

For trajectory evaluation, acceptable alternatives should be explicit too. Two agents may complete the same task using different valid tool sequences, so comparing against one canonical path can penalize a better strategy. Constraints on required effects, forbidden actions and resource use are a more durable contract than exact step-for-step matching.

[Nagappan S](https://hashnode.com/@imkarthik03)

[Oct 13](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac53cedbe529719bcce1bb7#reply-6acdd333d57024c379df1982)

Agreed on both. Versioning the dataset and keeping the previous release's cases as a frozen slice is what separates "the system got worse" from "the coverage got better". I'd report the stable slice and the full set side by side.

On trajectories, constraints are the better contract: required effects, forbidden actions and resource limits, rather than one canonical path. That way a better strategy isn't penalised just for taking a different route.

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 15](https://hashnode.com/posts/evals-how-do-you-actually-know-your-ai-feature-works/6ac48bb15d0e8169a118170c/comment/6ac53cedbe529719bcce1bb7#reply-6ad13cd5899f987201cfe481)

Reporting both sets also makes new-case regressions actionable: keep case identifiers stable and show which failures belong to the frozen slice versus the added coverage. Otherwise readers still have to infer whether the release broke old behavior or uncovered an older weakness.

For trajectory constraints, I would check the required effect in the resulting system state, not only the tool-call log. A plausible call sequence can still leave the task incomplete, while a shorter valid sequence may satisfy the same contract.
