Comment by Mike Dabydeen on "A Human in the Loop Does Not Automatically Make an AI Workflow Safe"

I'd measure how far the model's answer moves the reviewer, not only whether they override it. A reviewer who reads "Critical, 96%" before the report is anchored, and a low override rate can't tell you whether the reviewer or the anchor made the call.

One way to see it: route a small random sample, say 1 case in 20, to review with the model's fields hidden. The reviewer reads the report and system data and sets urgency alone. Then compare agreement with the model on blind cases against agreement on normal ones. If reviewers agree 95% of the time when they can see the model and 80% when they can't, that gap is the anchoring, and it says more about the review than the override rate does.

I'd also record the direction of each override. In your table, Medium raised to Critical and Critical lowered to High are different events. With a deterministic floor in place, a downgrade is the one that earns a second look, or at least a reason code picked from a short fixed list. Free-text reasons are hard to count later.

Override rate alone can mislead. A low rate could mean the model is performing well, or it could mean reviewers are anchoring on the model output.

I'll use the blind-sample idea. I'd split the evaluation into normal cases and blinded cases, then compare reviewer agreement and override behaviour between them.

I'll also track override direction explicitly. Medium to Critical tells us something different from Critical to High, especially when a deterministic safety floor is involved.

For reasons, a fixed taxonomy works better for analysis than free text alone. I can keep the free-text explanation for context and require a structured reason code in the audit trail.

That replaces "Did the human review it?" with a better question. Did the review process add independent signal beyond the model's recommendation?

Search Hashnode