Although on front-end you have to follow the flow, still there are lot of things which come at backend are not exactly backend stuff but sometime just something which needs to be done on server, so it is on you. It can be something that not all full stack developers are doing every day, but full time backend developers do. Sometime it is about scaling, sometime it is about caching strategy, sometime you need to in memory DBs or document based noSQL databases. Sometime, you have to deal with search engines like elastic search or it can be network related stuff if your product is in that domain. So in short, whatever you do in detail, there are lot of things involved. Despite of being full stack developer, I was mostly working on backend because of requirements and lack of such backend skills. Because if you are in PHP, you may need to know lot of different frameworks to survive if you are in service providing industry. Other than that, there are lot of algorithms that you also need to know for some projects.
There are lot of things which should be actually separate person job like machine learning etc. but in many cases, backend developer have to deal with it. So both sides have lot of stuff in it. So if you want to go in detail, choose one. Don't be framework specific, if on backend but try to understand concepts and methodologies.