Great post, the section on severity tag calibration really caught my eye. Usually, you spend way more time optimizing metrics like precision or F1-score, but you shouldn't overlook user alignment or the model's user experience. These are things you might not even consider at first.
Thanks!
Though calibration sits a step before the metrics — precision/F1 tell you whether a finding is correct, not whether "[blocker]" reads the same to a human as to the tool emitting it. A vocabulary problem, not a model one, and no score catches it.
Diokel
Hi, I'm Williams, a Software Engineering student, and I like to write about topics that interest me to research.
Great post, the section on severity tag calibration really caught my eye. Usually, you spend way more time optimizing metrics like precision or F1-score, but you shouldn't overlook user alignment or the model's user experience. These are things you might not even consider at first.