Your four moves verify a check before it gets the veto. I'd add one for after: keep proving it in production, because a ruler can go wrong later without anyone touching it. A change to the normalisation step, a new document template or a library upgrade can change what the hash sees, and the check keeps its authority the whole time.
The cheapest version is a small set of known-repetitive and known-varied documents fed through the live pipeline on a schedule, each with its expected verdict. If a planted near-duplicate comes back clean, the check loses its veto until someone looks. I'd also watch the veto rate itself. A check whose override count drops to zero after a deploy looks like quiet success, which is exactly the shape of failure you described.
On shadow mode: disagreement between the rule and the judge tells you how often they differ, not which one is right. A sampled human read of the disagreements is what settles the direction.
That fifth move is the one I should have included, and you've named why it matters: a ruler drifts without anyone touching it. A normalisation tweak, a new template, a library bump, and the hash quietly sees something different while the check keeps its full authority. The concrete version of your golden set is good precisely because it's cheap: fix the expected verdict for each known-repetitive and known-varied document, run them on every deploy, and alarm when a verdict changes that nobody intended. The check checking itself. And your point on shadow mode is the one people skip: the disagreement rate only tells you the rule and the judge differ, never which is right, so without a sampled human read of those disagreements you've measured noise, not direction.
Your four moves verify a check before it gets the veto. I'd add one for after: keep proving it in production, because a ruler can go wrong later without anyone touching it. A change to the normalisation step, a new document template or a library upgrade can change what the hash sees, and the check keeps its authority the whole time.
The cheapest version is a small set of known-repetitive and known-varied documents fed through the live pipeline on a schedule, each with its expected verdict. If a planted near-duplicate comes back clean, the check loses its veto until someone looks. I'd also watch the veto rate itself. A check whose override count drops to zero after a deploy looks like quiet success, which is exactly the shape of failure you described.
On shadow mode: disagreement between the rule and the judge tells you how often they differ, not which one is right. A sampled human read of the disagreements is what settles the direction.