---
title: "Comment by BotSailor on \"Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests\" | Hashnode"
description: "This is a really practical take on something that can get confusing very quickly with AI-powered applications. The part about flaky tests being an “assertion st"
url: "https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38410c1c1217cc41efe9f"
---

[Discussion](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda)

Comment [Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda)

[BotSailor](https://hashnode.com/@botsailor)

[Oct 5](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38410c1c1217cc41efe9f)

This is a really practical take on something that can get confusing very quickly with AI-powered applications.

The part about flaky tests being an “assertion strategy problem” really stood out to me. It’s tempting to test an LLM response the same way we would test a traditional function, but expecting the exact same wording from a model is setting the test up to fail.

I also liked the distinction between testing our application logic with mocks and testing the actual model behaviour separately. That feels like a much healthier approach than having the entire test suite depend on an external API being available and returning exactly what we expect.

The idea of asserting on structure and behaviour rather than exact generated text is probably something a lot of teams will learn the hard way. Great article, and very useful for anyone trying to move LLM features from a demo into production.

[Madhav Bhasin](https://hashnode.com/@madhavbhasin)

[Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38410c1c1217cc41efe9f#reply-6ac6242543b2dd81ee2ac8b9)

Thanks — "assertion strategy problem" is the reframe that made the rest of the testing approach click for us too. The demo-to-production gap on AI features is real and mostly comes down to exactly this: the habits that work for deterministic code silently fail on probabilistic output. Glad it was useful.