The EPS floor for labels missing from top_logprobs is another useful sensitivity check. Temperature scaling is exactly a logit rescaling for a complete probability distribution, but assigning 1e-6 to unreported options introduces an approximation before the fit.
I would report how often the gold label was unreported at each step and repeat the fit with a few plausible floors. If T changes substantially, it is partly compensating for probability truncation rather than only model calibration. That would qualify the result without changing any of the selected answers.
The NLL-vs-ECE argument is the strongest part: ECE jumping at bin edges makes it a bad search target, and the q = 0.92 table shows why NLL lands on the true rate. One thing I'd be curious about is the step-wise fit on only 729 right-group answers for the intent temperature. Since the intent step never sees wrong-group cases at fit time but is scaled on them at test time, would a held-out check on wrong-group answers alone show whether T=1.14 over- or under-corrects there? Also, since the middle bins stay >0.1 off, a two-parameter option like Platt/vector scaling or isotonic regression on the final product might be worth comparing, with a bootstrap interval on the test ECE to see whether 0.041→0.030 is outside the noise.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Since a prompt change means a refit and the calibration rerun takes about 3 minutes, I'd store each fitted T with the prompt version it came from and refuse to load one fitted on a different version. The intent step landing at 1.138 on BANKING77 and 0.619 on CLINC150 shows how far apart two fits can sit.