---
title: "Discussion on \"Your eval suite is green. Production is not.\" | Hashnode"
description: "Discussion on \"Your eval suite is green. Production is not.\". You ship an LLM feature. Three weeks later a Slack thread mentions that a customer got a strange answer. You dig in. The prompt has been edited twice since release, the provider has quietly re-quantis"
url: "https://hashnode.com/posts/your-eval-suite-is-green-production-is-not/6a5ffa89754869e55b3d2758"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Your eval suite is green. Production is not.](https://golikovichevhashnodedev.hashnode.dev/your-eval-suite-is-green-production-is-not?utm_source=hashnode&utm_medium=feed)

You ship an LLM feature. Three weeks later a Slack thread mentions that a customer got a strange answer. You dig in. The prompt has been edited twice since release, the provider has quietly re-quantis

Jul 21 · 2 min read · golikovichevhashnodedev.hashnode.dev  
Read the article

## Written by

[Mikhail Golikov](https://hashnode.com/@golikovichev)

@golikovichev · Hove

  
[View profile](https://hashnode.com/@golikovichev)

Software engineer building open-source Python testing tools (postman2pytest, secure-log2test, pytest-conversational). Test automation, pytest, CI. Background in QA automation. Hove, UK.

## Responses 1

Markdown supported · ⌘↵ to post

[Salman Parvez | ML Systems](https://hashnode.com/@ml-systems)

[Oct 27](https://hashnode.com/posts/your-eval-suite-is-green-production-is-not/6a5ffa89754869e55b3d2758/comment/6ae0db17fd614457ff92241b)

Turning every escalation into a concrete test is how good contractors handle callbacks. You can't predict every one, but you can make sure the same problem never brings you back twice. That's a habit we're building into ML Systems from the start.
