It's impossible to know everything that's going on in a codebase - not that different from work, even more so for an open source project where you aren't reviewing every single PR or know about each aspect or part of the code.
We try to have team leads for different packages/areas which are mostly determined based on interest/history github.com/babel/notes/blob/master/2017-04/april-….
We have to invest in getting more people involved so that we can get more reviewers ultimately.
You can find the time if it's (hopefully) fun enough to do, otherwise live your life with other hobbies, family, friends,et c.