Why has visual programming not caught on?

It seems like the concept of visual-based programming (Ex. VisualBASIC or drag/drop components that make code) keeps on reappearing albeit unsuccessfully both on front and back end. Why is that? It's pretty obvious that there is a persistent and strong demand for making programming easier and more accessible to those who have brilliant ideas but have no coding skills. I suspect that there is an elitist culture in which developers actually don't want things to be easy to protect their job security. After all, it makes sense that they would not dare spending months building something that will ultimately render their skills irrelevant. But are there other legitimate reasons that the visual programming concept doesn't work well?