I feel your pain. For what it's worth, I think I went through what you're going through in 2019. After many months of online classes and "independent study" I finally got to the point where I was coding activation functions and back-propagation. After all of that I understood why the bottleneck was data and compute, so I'd always be tied to some kind of API if I wanted to use anything that was relevant.
This made me accept and ultimately embrace ML as a black box. In the same way that I'm not familiar with the codebases of most of the tools I use every day in my coding career, I'm not going to be familiar with the ins and outs of ML. It's just another tool but it's a powerful one so it's important that I can use it effectively.
Because of that I'm very much in the API camp and as far as APIs go they seem to be pretty darn simple which is nice.