Especially lately, I am ditching all JS frameworks in favor of CSS3 (animation, transition,...) and VanillaJS. I only use small libraries, like reconnecting-websockets.js, which make my life easier. Why re-invent the wheel when I do not need an innovative solution but just something which works?
But careful there! I am not a professional front-end dev who builds the next Facebook. Depending on your project, things like Virtual Dom can make a huge difference, and when using Virtual Dom, you do need frameworks, like React; or HTML manipulation becomes tedious.
Another thing to look out for is big projects. We have Dart and TypeScript for a reason: To minimize stupid bugs. We are humans, so we have difficulties making sure that everything is exactly what we expect all the time. Though, given proficient developers, they will surely come up with an adequate solution in a timely fashion, written entirely in modern JS. But how many people could pull that off? What about mixed or loose teams?
So the bottomline is: We can go all the way with VanillaJS, but at some point, the performance becomes terrible or programming becomes tedious or we start making stupid bugs because we lose oversight, are missing knowledge or just have a bad day.