Depends on the kind of product that you are trying to make, and the amount of time your potential users can wait.
While UI libraries have their own shortcomings that you have highlighted. They also give you a some foundation to start from, and not necessarily start from scratch.
But if you were to start from scratch, You'd be able to get features, tailor-made to your requirements (which might be absent in UI libraries).