I really like your observation that you can't fully reason about these systems from the architecture alone, you have to run them and watch. I strongly agree with this.
That feels very different from traditional software engineering, where behavior is largely determined by logic we've explicitly written. With LLMs, the training objective may be simple, but the resulting behaviour can be surprisingly difficult to predict, especially once quantization, runtimes, context windows, and deployment constraints enter the picture.
I also agree that the understanding vs pattern-matching debate becomes less important when you're actually working with these models. At some point, the practical question shifts from "What should happen?" to "What actually happens?"
Appreciate the comment, and I'm glad the compression-of-knowledge and library analogies resonated with you. You can give my other articles a try, I am trying to write more about the interesting things about AI, apart from its technical usage, which we hear about in the corporate world like a mantra, haha. Really appreciate you dropping by.
The compression-of-knowledge framing is the part that finally made this click for me too. What I keep bumping into on the practical side is that the 'it somehow works' surprise cuts both ways. I run these models locally on modest hardware, and the same thing that makes next-word prediction feel like magic also makes them weirdly brittle: a model that should fly on paper will crawl the moment the runtime or the quantization is slightly off. The understanding vs pattern-matching debate honestly matters less to me than the fact that the behaviour is emergent enough that you cannot fully reason about it from the arcitecture alone, you have to run it and watch. Good writeup, the library analogy is a keeper.