Limin Ma

Thanks for the questions. Fitting the intent temperature on just the right-group answers is deliberate. It only needs to be honest when the group is right because that's what P(intent | group) measures. For wrong-group answers the right intent isn't among the options at all, so keeping their confidence low is the group step's job. Isotonic regression on the final confidence is in post 6 (test ECE 0.011), but the series doesn't try Platt or vector scaling. I ran the bootstrap you suggested to see whether the drop from 0.041 to 0.030 is outside the noise. The answer was less clear than I expected. With the post's T held fixed, the scaled ECE was lower in every resample of the test set. But when the calibration set is resampled too and T refitted each time, it's lower in only 88% of resamples. So the improvement is likely to hold but isn't certain, and most of the noise comes from fitting T on 770 answers.

Thanks for the careful read. Your check is worth running on any new dataset. When the right label is missing, the floor becomes its probability and can pull T up. I ran it on the post's two datasets. The good news is that the right label was never missing, and T doesn't change for floors from 1e-5 to 1e-10. Thanks again for spelling out the check so clearly.

Thanks, versioning T like that is a good safeguard. The thresholds in post 8 are searched on the scaled confidence, so they should be versioned along with it.

Thanks for reading. Post 8 is now up. It shows how the thresholds are chosen and how much the 98.7% could vary on other messages. One detail this post leaves out: the model needs a few minutes to answer the new calibration messages before the refit after a label change.

Thanks for the practical notes. Post 8 starts from the same three costs but turns them into two targets instead: actions should be right at least 98% of the time, and questions should offer the right intent at least 90% of the time. The result is what you predicted, with a high act threshold and 39.9% of test messages in the ask band. Minimizing expected cost directly is cleaner when a team can say roughly how the outcomes compare. Tracking weekly band shares is a nice addition. The post only suggests checking how often actions are right, which needs labels. Band shares don't, so they can flag drift sooner. On your last point: your tip on close pairs fills a gap in the post. When two intents are close, the confidence is at most about 0.5 and often lower. Many of these answers fall below the ask threshold of 0.493 and go to a human agent. Your rule would offer them the two options instead. I haven't tested it, so I can't say how many answers it would catch.