The review sampling policy becomes part of the training data once uncertain cases and disagreements are deliberately oversampled. That is useful for learning boundaries, but the resulting label distribution no longer represents ordinary traffic.

I would retain the sampling reason and, where possible, selection probability with each reviewed request. Keep a separate randomly sampled traffic slice for estimating auto-dispatch error and coverage at deployment prevalence. Otherwise a classifier can improve on the carefully curated hard-case set while the reported operating error moves simply because the review policy changed. The same distinction helps compare the cascade with a trained replacement fairly.