For the grass-or-not classifier, I would split the dataset by recording session or terrain section rather than randomly distributing adjacent camera frames. Otherwise training and validation can contain nearly identical views, and the small model can appear ready while failing on a different patch of ground or lighting condition.

The decision to upgrade hardware should follow that test, not just the observed frame rate. Log boundary cases such as grass beside gravel, shadows across the path, and a partially visible walkway. Those examples can reveal whether the task needs a different label or spatial output before it needs a larger accelerator. More compute cannot repair an ambiguous training target.