---
title: "Comment by Mike Dabydeen on \"My AI SRE said \"HIGH confidence\" and was wrong. What it took to make HIGH mean HIGH\" | Hashnode"
description: "On what to count: per run where any false hypothesis reaches HIGH, which you're already doing, with a confidence bound reported next to it, because zero in 29 r"
url: "https://hashnode.com/posts/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high/6afde240b1c9022055a97ab1/comment/6b003a1d51a88417780e2ad4"
---

[Discussion](https://hashnode.com/posts/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high/6afde240b1c9022055a97ab1)

Comment [My AI SRE said "HIGH confidence" and was wrong. What it took to make HIGH mean HIGH](https://hashnode.com/posts/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high/6afde240b1c9022055a97ab1)

[Mike Dabydeen](https://hashnode.com/@firelinks)

[2d ago](https://hashnode.com/posts/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high/6afde240b1c9022055a97ab1/comment/6b003a1d51a88417780e2ad4)

On what to count: per run where any false hypothesis reaches HIGH, which you're already doing, with a confidence bound reported next to it, because zero in 29 runs is a smaller claim than it looks. With no failures in n runs, the 95% upper bound on the true rate is roughly 3/n, so 29 clean runs still allows a false HIGH rate near 1 in 10. That's enough to show the challenge helped: at your old rate of 7 in 57, getting 29 clean runs in a row by chance is about a 2% event. To claim under 1 in 20 you'd need about 60 clean runs, and about 150 for 1 in 50.

That gives the larger measurement runs since a job. Add them to the 29 and publish the run count with the bound, and an on-call reader can see what HIGH is worth today. You'll also see it move when a new scenario or a model change shifts the rate, instead of finding out from a bad page at 3 am.
