One A/A pair gives a useful warning, but its observed gap is itself noisy. In your toy setup, two identical runs can happen to tie even though the same setup regularly produces four-point differences. Treating that zero as the floor would make the next small improvement look unusually convincing.
For a frequently used release gate, I would collect several unchanged pairs and report their gap distribution alongside the unstable cases. Keep the exact paired test for the candidate comparison, and treat the historical A/A distribution as a diagnostic rather than a guaranteed threshold inferred from one pair.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
One A/A pair gives a useful warning, but its observed gap is itself noisy. In your toy setup, two identical runs can happen to tie even though the same setup regularly produces four-point differences. Treating that zero as the floor would make the next small improvement look unusually convincing.
For a frequently used release gate, I would collect several unchanged pairs and report their gap distribution alongside the unstable cases. Keep the exact paired test for the candidate comparison, and treat the historical A/A distribution as a diagnostic rather than a guaranteed threshold inferred from one pair.