A wall-centered GradCAM map is a useful lead, but I would treat it as a hypothesis about the shortcut rather than confirmation by itself. The final convolutional map is coarse, and attribution does not directly measure what happens when that background changes.
Your counterfactual testing section could pair the same foreground gesture with several independently sampled backgrounds, then measure how often the prediction changes. A foreground-preserving control transformation would help separate background reliance from artifacts introduced by masking or compositing. If heatmap movement and prediction sensitivity agree across those pairs, the case for a targeted augmentation is much stronger.