A Valid Model Response Can Still Be an Unsafe One

I ran into this while building Maintenance Triage (link), a prototype that turns free-text hospital equipment reports into routing decisions. It is a working prototype and has not been deployed in a h