---
title: "Comment by Evelyn Wyatt on \"Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests\" | Hashnode"
description: "The schema-versus-value distinction is useful, but there’s an interesting gap between “valid output” and “correct output.” A response can satisfy the schema per"
url: "https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38d069811b96acdb1f7dd"
---

[Discussion](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda)

Comment [Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda)

[Evelyn Wyatt](https://hashnode.com/@Evelynwyatt)

[Oct 5](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38d069811b96acdb1f7dd)

The schema-versus-value distinction is useful, but there’s an interesting gap between “valid output” and “correct output.” A response can satisfy the schema perfectly while extracting the wrong invoice total, misclassifying a document, or producing a summary that misses a critical fact. I’d treat those as separate testing layers: deterministic tests verify the contract and how the application handles the response, while a smaller set of evaluation cases checks semantic correctness against known expectations. That also makes flaky tests easier to avoid without weakening coverage. The goal isn't to make every assertion deterministic; it's to make each test responsible for a property that can actually be judged reliably.

[Madhav Bhasin](https://hashnode.com/@madhavbhasin)

[Oct 7](https://hashnode.com/posts/testing-ai-dependent-code-mocking-llms-evaluating-outputs-and-avoiding-flaky-tests/6ac33dab2f76360ece243dda/comment/6ac38d069811b96acdb1f7dd#reply-6ac624071e7294a005eda001)

This is the cleaner framing than what the post uses. Contract tests verify the application handles the response correctly — schema, error paths, retry bounds. Evaluation cases verify the model extracted the right thing. Conflating the two is what produces tests that are either flaky or meaningless. "Make each test responsible for a property that can actually be judged reliably" is the principle the post should have opened with.
