Feed
Discussion
A Human in the Loop Does Not Automatically Make an AI Workflow Safe
One of the most common phrases in healthcare AI is: “A human is always in the loop.” It sounds reassuring. But a human reviewing an AI output does not automatically make the workflow safe. The impor
#ai#healthcare#machine-learning#software-engineering#ai-agents#testing#llm#large-language-models#healthtech
Responses
Markdown supported · ⌘↵ to post
I'd measure how far the model's answer moves the reviewer, not only whether they override it. A reviewer who reads "Critical, 96%" before the report is anchored, and a low override rate can't tell you whether the reviewer or the anchor made the call.
One way to see it: route a small random sample, say 1 case in 20, to review with the model's fields hidden. The reviewer reads the report and system data and sets urgency alone. Then compare agreement with the model on blind cases against agreement on normal ones. If reviewers agree 95% of the time when they can see the model and 80% when they can't, that gap is the anchoring, and it says more about the review than the override rate does.
I'd also record the direction of each override. In your table, Medium raised to Critical and Critical lowered to High are different events. With a deterministic floor in place, a downgrade is the one that earns a second look, or at least a reason code picked from a short fixed list. Free-text reasons are hard to count later.