Ran into almost the exact same wall with an intake classifier last year. The thing that finally broke the cycle for me was treating "Other" as a signal instead of a category — when confidence was low I routed it through a second pass that had the model summarize the email in one line and re-classify with that summary attached. Cut the dumping-ground effect dramatically. Also found that fewer, longer examples beat longer definitions; the model learned the category boundaries better from edge cases than from prose. Solid write-up — the debugging metaphor is exactly right.
Ran into almost the exact same wall with an intake classifier last year. The thing that finally broke the cycle for me was treating "Other" as a signal instead of a category — when confidence was low I routed it through a second pass that had the model summarize the email in one line and re-classify with that summary attached. Cut the dumping-ground effect dramatically. Also found that fewer, longer examples beat longer definitions; the model learned the category boundaries better from edge cases than from prose. Solid write-up — the debugging metaphor is exactly right.