You'll be fine in most cases for SEO.
Using JS, you limit yourself to only the Googlebot, as most search bots aren't capable of parsing it.
Beyond that, if you don't have static pages/routes, or you're not using pushState properly -- you're screwed. You'll be crawled, but when it comes time to find your site via the search, it'll be hard to rank high when only your root app URL is available. Make sure if you're linking to routes, your app is able to load the route in a separate new window (ie: yoursite.com/about-us), without going to the root of the app first and navigating to the page.
I'm not a fan of "hashbangs" for URLs either, since they aren't recognized by every service (like Google Analytics).
You also get better performance from SSR thanks to the way it renders the DOM. Rather than loading a fat JS bundle up front with all the compiled HTML, you serve the HTML first and hydrate the DOM with JS interactivity. Makes React faster since it's bogged down less with all the HTML rendering, and makes your server faster since it can offset loading of often less critical assets (like JS). You can see from these infographics the logic behind the performance benefit:
Obviously, things like SSR matter for apps that plan optimizing for performance and SEO. If you're making an app that doesn't need to be Google searched, or doesn't have to be lightning fast -- client side rendering works just fine if you do it right.