What It Takes to Trust an AI Deployment Gate: Measuring Four Models Against a Dumb Baseline

Thanks for this Kartik, it's a really good point. You're right that the artefact can settle it without trusting the commit message. Deploy history lives outside the repo, so the change can't fake it.

To your question, I'd start with context. Telling the model "this restores the exact version that was running before the deploy 36 minutes ago", verified from deploy history, is a far stronger than a commit message claiming it's a revert. I'd probably check that the result matches the previously running version rather than just comparing diffs, and I can measure whether it's enough with the existing harness, alongside a fake-revert scenario to make sure it can't be gamed.

If the models still halt it with the context, I'd then consider using a rule, but a narrow one: a verified rollback can move from halt to canary and no further. The floor would still only ever raise risk. This would be a separate, bounded exception that gets the fix to a slice of traffic quickly, with a human still in the loop.

If you'd like to try it, the scenario is in the repo as revert_of_a_bad_deploy, and I'd welcome a PR.