The leave-one-season-out call is the right instinct. With five seasons the failure mode is exactly what you named: sharp on training data, dead on a season it never saw. I would also slice-check the rare states, since an 8-state space can hide a class showing up only a few times. Did any state end up too sparse to trust?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
The leave-one-season-out call is the right instinct. With five seasons the failure mode is exactly what you named: sharp on training data, dead on a season it never saw. I would also slice-check the rare states, since an 8-state space can hide a class showing up only a few times. Did any state end up too sparse to trust?