You can keep your frontend state in the form input. It can be simple hidden input with json data inside it. And when you do request, you can use value of the input to restore/modify state of your block.

Yes, visibility widgets, even when hidden - stays in widget tree, but you can set replacement widget, by default its SizedBox.shrink(). In my case I didn't want if/else in that part of code and used Visibility()