I really like your observation that you can't fully reason about these systems from the architecture alone, you have to run them and watch. I strongly agree with this.

That feels very different from traditional software engineering, where behavior is largely determined by logic we've explicitly written. With LLMs, the training objective may be simple, but the resulting behaviour can be surprisingly difficult to predict, especially once quantization, runtimes, context windows, and deployment constraints enter the picture.

I also agree that the understanding vs pattern-matching debate becomes less important when you're actually working with these models. At some point, the practical question shifts from "What should happen?" to "What actually happens?"

Appreciate the comment, and I'm glad the compression-of-knowledge and library analogies resonated with you. You can give my other articles a try, I am trying to write more about the interesting things about AI, apart from its technical usage, which we hear about in the corporate world like a mantra, haha. Really appreciate you dropping by.