Comment by Ahmet Özel on "Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests"

Ahmet Özel

AI Engineer. Computer Vision, RAG and LLM agents.

The invoice example seems to call for normalized value comparison rather than removing the value assertion. '1,234.56' and '1234.56' can represent the same amount, but a schema-valid positive number could still be the wrong total, and the real-API test shown would accept it.

I would parse the amount under an explicit locale and currency contract, then compare it with the labelled invoice amount. Keep the client mocks for malformed output and application behavior, and keep a separate extraction-quality suite with known source documents. That preserves the useful deterministic test boundary while ensuring that eliminating formatting flakiness does not also eliminate the test's ability to detect an incorrect extraction.

Madhav Bhasin

Production patterns for backend engineers who are tired of hello-world tutorials.

Oct 7

You're right that schema validity and extraction correctness are different things, and the post conflates them. Asserting total_amount > 0 catches malformed output, not wrong output. The locale-aware comparison against a labelled expected value is the correct approach for testing extraction quality — it just belongs in the separate evaluation suite against known source documents, not the mocked unit tests. The mocked tests cover application behaviour; the evaluation suite covers whether the model extracted the right value. Both need to exist.

That separation also makes a deliberately wrong, schema-valid amount a useful control. A mocked response with the wrong total should exercise the application path without being mistaken for evidence that extraction quality is good. The labelled-document suite should then fail that same value under the locale and currency contract.

Keeping a small set of such negative controls alongside the formatting equivalents would demonstrate that the suite remains sensitive to semantic errors after normalization, rather than merely becoming quieter.

Search Hashnode