Discussion on "Diagnosing Shortcut Learning in Spatial Vision with XAI Heatmaps"

Akhila Sharon N. · 2026-10-08T12:05:04.419Z

You build a computer vision model for spatial gesture recognition. The training loss drops smoothly, validation accuracy hits 98%, and everything looks solid in notebook evaluation. Then you test it i

A wall-centered GradCAM map is a useful lead, but I would treat it as a hypothesis about the shortcut rather than confirmation by itself. The final convolutional map is coarse, and attribution does not directly measure what happens when that background changes.

Your counterfactual testing section could pair the same foreground gesture with several independently sampled backgrounds, then measure how often the prediction changes. A foreground-preserving control transformation would help separate background reliance from artifacts introduced by masking or compositing. If heatmap movement and prediction sensitivity agree across those pairs, the case for a targeted augmentation is much stronger.

Completely agree. GradCAM is a hypothesis rather than confirmation on its own, especially given its spatial coarseness. Running foreground-preserving control transformations with independently sampled backgrounds is the right way to avoid confounding artefacts and validate the shortcut. Appreciate the detailed feedback.

The foreground-preserving control is especially useful when compositing creates a visible edge around the hand. I would compare prediction changes against that control before interpreting them as reliance on the wall or room.

Keeping the original foreground identities together when splitting the evaluation set also matters: otherwise the same gesture cutout can appear in both development and held-out pairs. Reporting sensitivity separately for seen and unseen backgrounds would make the augmentation result easier to interpret.

Search Hashnode

Diagnosing Shortcut Learning in Spatial Vision with XAI Heatmaps

Responses3