One of the biggest problems you have with premature optimization is that you will never be done. You write some code, see how you can optimize it, then write the optimization code. Then you find a way to optimize the optimization code, so you write some code optimization optimization code. But then you come up with some way to optimize that. However, while writing the code optimization optimization code optimization code, you find out, that you are de-optimizing something you optimized before, so you have to start compromising between optimizations.

After spending weeks on a task which could have been finished in a matter of hours, you finally deliver a product, which is over-engineered (as in "no one else can maintain it"), probably even less performant than without all the "optimizations", because you optimized the wrong thing at an expense of literally everything else, and very likely hard if not impossible to extend with new code and features.
Optimizations can be very useful, but only if they are needed. First create the product and check if you actually need to optimize something. If it works and there are no problems, why pour time into optimizations?