---
title: "Comment by Arnold Holm on \"Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests\" | Hashnode"
description: "The client-level fixture can also distinguish a truncated response from malformed JSON. Add a response with finish_reason set to length, then check that extract"
url: "https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac3a9a510e928a5044df243"
---

[Discussion](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda)

Comment [Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda)

[Oct 5](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac3a9a510e928a5044df243)

The client-level fixture can also distinguish a truncated response from malformed JSON. Add a response with finish\_reason set to length, then check that extraction stops before saving a partial result and that any retry is bounded. A valid-looking prefix should not cross the same persistence boundary as a complete response.

[Madhav Bhasin](https://hashnode.com/@madhavbhasin)

[Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac3a9a510e928a5044df243#reply-6ac6238d28eaabc1974c0380)

Solid edge case — finish\_reason="length" is exactly the kind of failure that looks like success if you're only checking that the response came back. Checking it before any persistence call is the right place to gate it, and worth adding to the fixture suite alongside the malformed JSON case. The bounded retry matters too — a truncated response retried with the same prompt and same token limit will truncate again.

[Arnold Holm](https://hashnode.com/@stratcorealpha)

[Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac3a9a510e928a5044df243#reply-6ac63b619fa0a3660c75abc3)

Yes, the unchanged token limit makes that retry a repeat of the same failure. I would make the length case return a clear incomplete-result state, then check that no save occurs and that the retry limit is reached. Increasing the allowance should be a separate explicit choice, rather than an endless retry.