On what to count: per run where any false hypothesis reaches HIGH, which you're already doing, with a confidence bound reported next to it, because zero in 29 runs is a smaller claim than it looks. With no failures in n runs, the 95% upper bound on the true rate is roughly 3/n, so 29 clean runs still allows a false HIGH rate near 1 in 10. That's enough to show the challenge helped: at your old rate of 7 in 57, getting 29 clean runs in a row by chance is about a 2% event. To claim under 1 in 20 you'd need about 60 clean runs, and about 150 for 1 in 50.
That gives the larger measurement runs since a job. Add them to the 29 and publish the run count with the bound, and an on-call reader can see what HIGH is worth today. You'll also see it move when a new scenario or a model change shifts the rate, instead of finding out from a bad page at 3 am.
Mike Dabydeen
On what to count: per run where any false hypothesis reaches HIGH, which you're already doing, with a confidence bound reported next to it, because zero in 29 runs is a smaller claim than it looks. With no failures in n runs, the 95% upper bound on the true rate is roughly 3/n, so 29 clean runs still allows a false HIGH rate near 1 in 10. That's enough to show the challenge helped: at your old rate of 7 in 57, getting 29 clean runs in a row by chance is about a 2% event. To claim under 1 in 20 you'd need about 60 clean runs, and about 150 for 1 in 50.
That gives the larger measurement runs since a job. Add them to the 29 and publish the run count with the bound, and an on-call reader can see what HIGH is worth today. You'll also see it move when a new scenario or a model change shifts the rate, instead of finding out from a bad page at 3 am.