Good distinction between system failure and eval failure. The next thing I’d want is to make the eval itself a versioned artifact.
Each run should carry the fixture version, case type, scorer version, scoring dimensions, judge configuration if an LLM judge is used, and a small human-audited slice. Otherwise the score can look precise while the thing being measured keeps changing.
That also makes bad evals easier to debug. You can ask whether the system regressed, the case set is unrepresentative, the scorer is applying the wrong criterion, or the production workload drifted away from the fixture set.
Ken
Good distinction between system failure and eval failure. The next thing I’d want is to make the eval itself a versioned artifact.
Each run should carry the fixture version, case type, scorer version, scoring dimensions, judge configuration if an LLM judge is used, and a small human-audited slice. Otherwise the score can look precise while the thing being measured keeps changing.
That also makes bad evals easier to debug. You can ask whether the system regressed, the case set is unrepresentative, the scorer is applying the wrong criterion, or the production workload drifted away from the fixture set.