This is a really practical take on something that can get confusing very quickly with AI-powered applications.

The part about flaky tests being an “assertion strategy problem” really stood out to me. It’s tempting to test an LLM response the same way we would test a traditional function, but expecting the exact same wording from a model is setting the test up to fail.

I also liked the distinction between testing our application logic with mocks and testing the actual model behaviour separately. That feels like a much healthier approach than having the entire test suite depend on an external API being available and returning exactly what we expect.

The idea of asserting on structure and behaviour rather than exact generated text is probably something a lot of teams will learn the hard way. Great article, and very useful for anyone trying to move LLM features from a demo into production.