Feed
Discussion
Rose Alvina
Being upfront about the trade is what makes this series credible: 86.9% on the BANKING77 test set, compared against 83 to 85% for classifiers trained on 10 messages per intent and about 93% with full training data. Framing the goal as knowing which answers to act on, which to clarify and which to hand off is a practical way to think about it. Refitting the calibration in seconds when labels change, instead of retraining, also keeps the setup cheap to maintain.
Nice framing of act / clarify / hand off as the real output. A practical note for anyone applying this to a support bot: pick the two thresholds from costs rather than accuracy. Estimate what a wrong automatic action costs versus a clarifying question versus a human handoff, then choose the act and hand-off cutoffs on a held-out set that minimise total expected cost. That usually lands you on a high "act" threshold and a surprisingly wide "clarify" band. Also watch calibration drift: track the share of traffic in each band per week. If the "act" share drops suddenly, new intents or wording have appeared, and that is the moment to label a fresh batch and refit, which as you note takes seconds. Finally, when an intent's top two labels are close in probability, a clarifying question that names both options tends to resolve it in one turn.