--- title: "Discussion on \"Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests\" | Hashnode" description: "Discussion on \"Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests\". How to test FastAPI services that call LLMs — client-level mocking, schema validation over exact assertions, prompt snapshots" url: "https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda" --- [Feed](https://hashnode.com/feed) / Discussion # [Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests](https://blog.madhav.dev/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests?utm_source=hashnode&utm_medium=feed) How to test FastAPI services that call LLMs — client-level mocking, schema validation over exact assertions, prompt snapshots Oct 5 · 6 min read · blog.madhav.dev Read the article - [#fastapi](https://hashnode.com/tag/fastapi) - [#python](https://hashnode.com/tag/python) - [#pytest](https://hashnode.com/tag/pytest) - [#llm](https://hashnode.com/tag/llm) - [#backend-engineering](https://hashnode.com/tag/backend-engineering) - [#ai](https://hashnode.com/tag/ai) ## Written by [Madhav Bhasin](https://hashnode.com/@madhavbhasin) @madhavbhasin · Noida, India [View profile](https://hashnode.com/@madhavbhasin) Production patterns for backend engineers who are tired of hello-world tutorials. Senior full-stack engineer and engineering lead with 5+ years building cloud-native systems at scale. I write about the things the docs don't cover — GCP, FastAPI, Terraform, event-driven architecture, and AI-powered backends. No hello-world demos. ## Responses 16 Markdown supported · ⌘↵ to post [Rishita Sharma](https://hashnode.com/@narawaybuilder) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac5cd104ca22d4f001e92fc) The invoice total example is exactly how my CI got ignored too. One formatting change from the model and suddenly everyone just reruns until green. Switching to schema checks plus a small normalizer for numbers and dates fixed most of it for me. The part I still struggle with is free text fields. Do you assert anything on those beyond length and required keywords? [Madhav Bhasin](https://hashnode.com/@madhavbhasin) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac5cd104ca22d4f001e92fc#reply-6ac6233528eaabc1974c0325) The normaliser for numbers and dates is exactly the right move — one fix, whole category gone. For free text: three things I actually assert on — required concept is present (loose, not keyword), length is within a sane range, and no model artefacts (leading "Sure!", markdown fences in plain-text fields). Beyond that I treat it as an evaluation problem, not a unit test problem. Weekly real API runs are where those quality regressions surface, not mocked tests. [Rishita Sharma](https://hashnode.com/@narawaybuilder) [Oct 8](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac5cd104ca22d4f001e92fc#reply-6ac722b0fb0d064306ade49d) Treating free text as an evaluation problem instead of a unit test makes sense. The artefact check is a good one, I hadn't thought about asserting on a leading "Sure!". Weekly real runs sounds like the right cadence to me. [netproxy](https://hashnode.com/@argumentmoney8117) [Oct 5](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac42d3e350baa25b6591b81) Good breakdown of the tradeoffs. One middle ground worth mentioning: for the live-API integration tests, passing a fixed seed (where the provider supports it) together with temperature 0 gets you much closer to deterministic without going fully mocked. It's not a guarantee - the same seed can still drift across model versions - but it noticeably reduces the retry count. Pairs well with the normalization point in the comments. [Madhav Bhasin](https://hashnode.com/@madhavbhasin) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac42d3e350baa25b6591b81#reply-6ac6236edf1531d125998a5c) Good addition — temperature=0 plus a fixed seed is a useful middle ground, especially for integration tests where you want some real-API coverage but less noise. The caveat you flagged is the right one: seed behaviour isn't guaranteed across model versions, so it's a noise reducer not a flakiness eliminator. Still worth doing. I'd pair it with snapshotting the model version in the test so a version bump is a visible signal rather than a silent source of drift. [Arnold Holm](https://hashnode.com/@stratcorealpha) [Oct 5](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac3a9a510e928a5044df243) The client-level fixture can also distinguish a truncated response from malformed JSON. Add a response with finish\_reason set to length, then check that extraction stops before saving a partial result and that any retry is bounded. A valid-looking prefix should not cross the same persistence boundary as a complete response. [Madhav Bhasin](https://hashnode.com/@madhavbhasin) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac3a9a510e928a5044df243#reply-6ac6238d28eaabc1974c0380) Solid edge case — finish\_reason="length" is exactly the kind of failure that looks like success if you're only checking that the response came back. Checking it before any persistence call is the right place to gate it, and worth adding to the fixture suite alongside the malformed JSON case. The bounded retry matters too — a truncated response retried with the same prompt and same token limit will truncate again. [Arnold Holm](https://hashnode.com/@stratcorealpha) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac3a9a510e928a5044df243#reply-6ac63b619fa0a3660c75abc3) Yes, the unchanged token limit makes that retry a repeat of the same failure. I would make the length case return a clear incomplete-result state, then check that no save occurs and that the retry limit is reached. Increasing the allowance should be a separate explicit choice, rather than an endless retry. [Ahmet Özel](https://hashnode.com/@ahmetozel) [Oct 5](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac39aaa147f64827744312d) The invoice example seems to call for normalized value comparison rather than removing the value assertion. '1,234.56' and '1234.56' can represent the same amount, but a schema-valid positive number could still be the wrong total, and the real-API test shown would accept it. I would parse the amount under an explicit locale and currency contract, then compare it with the labelled invoice amount. Keep the client mocks for malformed output and application behavior, and keep a separate extraction-quality suite with known source documents. That preserves the useful deterministic test boundary while ensuring that eliminating formatting flakiness does not also eliminate the test's ability to detect an incorrect extraction. [Madhav Bhasin](https://hashnode.com/@madhavbhasin) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac39aaa147f64827744312d#reply-6ac623c86fa83e597a3f61a5) You're right that schema validity and extraction correctness are different things, and the post conflates them. Asserting total\_amount > 0 catches malformed output, not wrong output. The locale-aware comparison against a labelled expected value is the correct approach for testing extraction quality — it just belongs in the separate evaluation suite against known source documents, not the mocked unit tests. The mocked tests cover application behaviour; the evaluation suite covers whether the model extracted the right value. Both need to exist. [Ahmet Özel](https://hashnode.com/@ahmetozel) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac39aaa147f64827744312d#reply-6ac654bfa69e92b74b22dd20) That separation also makes a deliberately wrong, schema-valid amount a useful control. A mocked response with the wrong total should exercise the application path without being mistaken for evidence that extraction quality is good. The labelled-document suite should then fail that same value under the locale and currency contract. Keeping a small set of such negative controls alongside the formatting equivalents would demonstrate that the suite remains sensitive to semantic errors after normalization, rather than merely becoming quieter. [Evelyn Wyatt](https://hashnode.com/@Evelynwyatt) [Oct 5](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38d069811b96acdb1f7dd) The schema-versus-value distinction is useful, but there’s an interesting gap between “valid output” and “correct output.” A response can satisfy the schema perfectly while extracting the wrong invoice total, misclassifying a document, or producing a summary that misses a critical fact. I’d treat those as separate testing layers: deterministic tests verify the contract and how the application handles the response, while a smaller set of evaluation cases checks semantic correctness against known expectations. That also makes flaky tests easier to avoid without weakening coverage. The goal isn't to make every assertion deterministic; it's to make each test responsible for a property that can actually be judged reliably. [Madhav Bhasin](https://hashnode.com/@madhavbhasin) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38d069811b96acdb1f7dd#reply-6ac624071e7294a005eda001) This is the cleaner framing than what the post uses. Contract tests verify the application handles the response correctly — schema, error paths, retry bounds. Evaluation cases verify the model extracted the right thing. Conflating the two is what produces tests that are either flaky or meaningless. "Make each test responsible for a property that can actually be judged reliably" is the principle the post should have opened with. [BotSailor](https://hashnode.com/@botsailor) [Oct 5](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38410c1c1217cc41efe9f) This is a really practical take on something that can get confusing very quickly with AI-powered applications. The part about flaky tests being an “assertion strategy problem” really stood out to me. It’s tempting to test an LLM response the same way we would test a traditional function, but expecting the exact same wording from a model is setting the test up to fail. I also liked the distinction between testing our application logic with mocks and testing the actual model behaviour separately. That feels like a much healthier approach than having the entire test suite depend on an external API being available and returning exactly what we expect. The idea of asserting on structure and behaviour rather than exact generated text is probably something a lot of teams will learn the hard way. Great article, and very useful for anyone trying to move LLM features from a demo into production. [Madhav Bhasin](https://hashnode.com/@madhavbhasin) [Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38410c1c1217cc41efe9f#reply-6ac6242543b2dd81ee2ac8b9) Thanks — "assertion strategy problem" is the reframe that made the rest of the testing approach click for us too. The demo-to-production gap on AI features is real and mostly comes down to exactly this: the habits that work for deterministic code silently fail on probabilistic output. Glad it was useful.