There are two questions here - SEO and performance and I'll answer them separately. 🙂
Should you SSR: Probably, but not for the reasons you think
But... that being said, the Google bot in particular is a PhantomJS script (if I remember correctly) that runs as Chrome 41. That means that you need add a lot of polyfills for things you take for granted, like native Promises and animations. This is why probably SSR makes more sense since you won't be shipping down unnecessary polyfills to all your users just so that the Google bot can render your page.
Here is a pretty neat video on this, straight from Google:
Should you SSR: I don't think so.
With tiny libraries like Preact and VueJS, I don't feel performance will take a huge hit. Bundle sizes are getting smaller by the day, so I don't feel you should take the effort to SSR here.
However, if your bundle size is huge, then SSR kind of makes sense.
Should you SSR: Definitely
I know it's not part of your original question, but I feel like it's a point worth mentioning. Accessibility takes a pretty big hit if you don't SSR.
There are compressing browsers like Opera mini, as well as web views that don't execute JS the way you want it to, or sometimes not execute JS at all. In cases like these, SSR is the only way you can deliver content to your user(s).
I've personally worked at a company where ~30% of our user base was on either Opera mini or UC Browser - both of which are compressing browsers that are notorious for not executing JS the way you want it to. We had to take SSR very seriously there, so as to keep those users.
Food for thought. 😄