---
title: "Discussion on \"My AI SRE said \"HIGH confidence\" and was wrong. What it took to make HIGH mean HIGH\" | Hashnode"
description: "Discussion on \"My AI SRE said \"HIGH confidence\" and was wrong. What it took to make HIGH mean HIGH\". The one word an on-call engineer reads first in an AI root-cause report is the confidence. At 3 am, \"HIGH\" means \"act on this\". So the most important question about any AI SRE tool is not how often it"
url: "https://hashnode.com/posts/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high/6afde240b1c9022055a97ab1"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [My AI SRE said "HIGH confidence" and was wrong. What it took to make HIGH mean HIGH](https://alertigate.hashnode.dev/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high?utm_source=hashnode&utm_medium=feed)

The one word an on-call engineer reads first in an AI root-cause report is the confidence. At 3 am, "HIGH" means "act on this". So the most important question about any AI SRE tool is not how often it

4d ago · 9 min read · alertigate.hashnode.dev  
Updated 2d ago  
Read the article

-   [#sre](https://hashnode.com/tag/sre)
-   [#kubernetes](https://hashnode.com/tag/kubernetes)
-   [#ai](https://hashnode.com/tag/ai)
-   [#observability](https://hashnode.com/tag/observability)
-   [#devops](https://hashnode.com/tag/devops)

## Written by

[Feres Berbeche](https://hashnode.com/@feres)

@feres

  
[View profile](https://hashnode.com/@feres)

https://alertigate.com/

## Responses 1

Markdown supported · ⌘↵ to post

[Mike Dabydeen](https://hashnode.com/@firelinks)

[2d ago](https://hashnode.com/posts/my-ai-sre-said-high-confidence-and-was-wrong-what-it-took-to-make-high-mean-high/6afde240b1c9022055a97ab1/comment/6b003a1d51a88417780e2ad4)

On what to count: per run where any false hypothesis reaches HIGH, which you're already doing, with a confidence bound reported next to it, because zero in 29 runs is a smaller claim than it looks. With no failures in n runs, the 95% upper bound on the true rate is roughly 3/n, so 29 clean runs still allows a false HIGH rate near 1 in 10. That's enough to show the challenge helped: at your old rate of 7 in 57, getting 29 clean runs in a row by chance is about a 2% event. To claim under 1 in 20 you'd need about 60 clean runs, and about 150 for 1 in 50.

That gives the larger measurement runs since a job. Add them to the 29 and publish the run count with the bound, and an on-call reader can see what HIGH is worth today. You'll also see it move when a new scenario or a model change shifts the rate, instead of finding out from a bad page at 3 am.
