If not used properly, these libraries do become a hinderance.
Developers usually opt for frameworks for the ease of development, but end up without using most of the features they provided. Many times a framework ends up becoming a dead weight in the project; which you can neither remove, nor put it to efficient use.
IMO, it is only okay to use a UI framework when:
Except for the above two cases, there's always an other, better way to handle UI in a project. Every developer has his own way of coding and you might not prefer the way a framework approaches something.
"Frameworks are opinionated, and opinions constrain flexibility."
Personally, I like to start with just a CSS reset, and some basic styles (which, you could say, is a little framework of my own, in a way).
That said, frameworks can always be taken as an inspiration. They establish good guidelines that'd help you in structuring your markup and CSS.
A good idea is to take parts of these frameworks that are relevant to your project (say, grid system) and leave the rest of it.
Keep the good ideas, and leave the bad/unnecessary ones. 🙂