---
title: "Discussion on \"356 of the 528 eval cases we could judge have never failed. Our pass rate could only move 21 points.\n\" | Hashnode"
description: "Discussion on \"356 of the 528 eval cases we could judge have never failed. Our pass rate could only move 21 points.\n\". TL;DR: I broke out per-case results for the incident-harvested part of our eval suite, 611 of our 1,400 cases, and asked which cases have ever discriminated between two shipped versions. Of the 528 wi"
url: "https://hashnode.com/posts/356-of-the-528-eval-cases-we-could-judge-have-never-failed-our-pass-rate-could-only-move-21-points/6a7e1b263c3b97238af3dfed"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [356 of the 528 eval cases we could judge have never failed. Our pass rate could only move 21 points.](https://ethanwalkerwrites.hashnode.dev/356-of-the-528-eval-cases-we-could-judge-have-never-failed-our-pass-rate-could-only-move-21-points?utm_source=hashnode&utm_medium=feed)

TL;DR: I broke out per-case results for the incident-harvested part of our eval suite, 611 of our 1,400 cases, and asked which cases have ever discriminated between two shipped versions. Of the 528 wi

Aug 13 · 8 min read · ethanwalkerwrites.hashnode.dev  
Read the article

## Written by

[ethanwalker](https://hashnode.com/@ethanwritesai)

@ethanwritesai · San Francisco, USA

  
[View profile](https://hashnode.com/@ethanwritesai)

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
