SVGs definitely work really well on the web but yeah, there are few issues you would face depending on what you're trying to do.
Code bloat
There's a lot of code bloat while using inline SVGs. Some SVGs with a lot of points can take up many lines of code. This makes your code look messy and you'd also face lot of trouble with code editors like atom when they freeze cause of the code bloat.
External referencing issue
One great thing with SVGs is that you can dump all your svg code into a single .svg file and reference them with the code using the <use> tag(kinda like an image sprite). I've also written a story on this if you want to dive deep. :)
Now, referencing is pretty great feature and solves our problem of code bloat but here's a catch: You can only reference SVGs if they are on the same url. This means that you cannot put your SVGs onto a CDN. You can check the bug here
Dimensions issue with referencing
So this is a weird issue with referencing again, when you reference an svg using the <use> tag, the dimensions that come out are different than what you'd get when you directly use the SVG. For fixing this, you'd have to specify the dimensions again in your CSS.
viewBox Manipulation issue
viewBox attribute in SVG basically provides a region for our SVG content do display in. viewBox being an attribute, can be manipulated by Javascript but there's a serious need of a viewBox property in CSS. Here is an in-depth article on Why We Need A viewBox Property in CSS.
Now there definitely are more problems with SVGs, these were just a few ones that I've faced till now.
This however does not mean that they are any less powerful, all these problems can be solved and there's always room for enhancements.
Hope it helps! 🙂