Comment by Kartik N V J K on "How a 75% Skill evaluation can hide a regression"

CommentHow a 75% Skill evaluation can hide a regression

Kartik N V J K

AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor

This is the failure mode aggregate scores are built to hide, both runs read 75% while one assertion flipped FAIL to PASS and another went the other way, net zero on the dashboard, real regression underneath. I got paged at 3 AM by exactly this, green CI, regression still live, and wrote up the per-assertion diffing I switched to: kartiknvjk.hashnode.dev/my-ci-evals-were-green-a-…. Do you alert on per-assertion deltas, or is the single score still the gate?

JH

Jason Huang

I maintain practical tools for agent workflows and evidence review.

Oct 1

Thanks, Kartik. I maintain the checker described here. It reports per-assertion changes from completed evaluation records; it does not send CI alerts or decide whether to ship. The 75% example is synthetic, so a flagged change is something to inspect alongside the original outputs and grading reliability, rather than proof of a repeatable regression.

Search Hashnode