The simplest way is to use Next.js. It's a framework that let's you build SSR (server side rendered) React apps. I really like how far it's progressed, and provides enough customization that should cover most/all of your needs.
Doing it manually ends up being a lot of work, but if you require it then there are many tutorials and boilerplates on building universal React apps. Here's a Medium article I found that seems to be good.