Comment by Rishita Sharma on "Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests"

Home
Feed

Extend

Hashnode Pro
Hashnode for Agents

Resources

The foreword
Hashnode gql skill

Buildful - AI coding agent that delivers frontier-level results at a fraction of the cost Bug0 - The AI-native e2e QA regression testing Hackathons Changelog Brand @hashnode on X Hashnode on LinkedIn Support - hello+support@hashnode.com Code of Conduct Terms Privacy Sitemap

Search Hashnode

Search posts, tags, users, and pages

Comment by Rishita Sharma on "Testing AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests" | Hashnode

Discussion

CommentTesting AI-Dependent Code: Mocking LLMs, Evaluating Outputs, and Avoiding Flaky Tests

Rishita Sharma

Sharing practical lessons from building Naraway and shipping useful products.

Oct 7

The invoice total example is exactly how my CI got ignored too. One formatting change from the model and suddenly everyone just reruns until green. Switching to schema checks plus a small normalizer for numbers and dates fixed most of it for me. The part I still struggle with is free text fields. Do you assert anything on those beyond length and required keywords?

Madhav Bhasin

Production patterns for backend engineers who are tired of hello-world tutorials.

Oct 7

The normaliser for numbers and dates is exactly the right move — one fix, whole category gone.

For free text: three things I actually assert on — required concept is present (loose, not keyword), length is within a sane range, and no model artefacts (leading "Sure!", markdown fences in plain-text fields). Beyond that I treat it as an evaluation problem, not a unit test problem. Weekly real API runs are where those quality regressions surface, not mocked tests.

Treating free text as an evaluation problem instead of a unit test makes sense. The artefact check is a good one, I hadn't thought about asserting on a leading "Sure!". Weekly real runs sounds like the right cadence to me.

Rishita Sharma

Sharing practical lessons from building Naraway and shipping useful products.

Oct 8