Kevin Kirui

Override rate alone can mislead. A low rate could mean the model is performing well, or it could mean reviewers are anchoring on the model output. I'll use the blind-sample idea. I'd split the evaluation into normal cases and blinded cases, then compare reviewer agreement and override behaviour between them. I'll also track override direction explicitly. Medium to Critical tells us something different from Critical to High, especially when a deterministic safety floor is involved. For reasons, a fixed taxonomy works better for analysis than free text alone. I can keep the free-text explanation for context and require a structured reason code in the audit trail. That replaces "Did the human review it?" with a better question. Did the review process add independent signal beyond the model's recommendation?

Testing what the reviewer actually sees, separately from the model output, is the part I'm taking from this. Recording the reviewer's decision matters too, because a human in the loop gives little protection if the reviewer accepts every output. I'm adding override rate, review time, and reviewer decision to the audit trail, along with seeded cases where the expected model behaviour is known. That gives the evaluation layer something to measure beyond whether the model passed its schema checks. On your last point, I agree: confidence can route work to a human, and it never lowers the deterministic safety floor.

Agreed on the inline versus async split. I'm limiting inline tests to the ones that can block or constrain downstream action: schema validity, required fields, safety-floor violations, and known injection patterns. The heavier behavioural tests run asynchronously against logged cases, since many of them need reviewer outcomes or longitudinal workflow data. Your point about confidence versus workflow authority holds too. Confidence can decide whether a case needs human review. It never gives the model authority to bypass a deterministic constraint. I'm treating that separation as a core design rule for the system.