---
title: "Comment by JustOneCX on \"Evaluation Isn't One Number: Lessons from a Microservice Fault Classifier\" | Hashnode"
description: "The 0.80 versus 0.57 gap is the number that matters most. Did you hold out whole systems for the test set, or random windows from systems the model already saw?"
url: "https://hashnode.com/posts/evaluation-isn-t-one-number-lessons-from-a-microservice-fault-classifier/6aff4baec27e45f5b21f553d/comment/6b01dcb21412d11724da80e5"
---

[Discussion](https://hashnode.com/posts/evaluation-isn-t-one-number-lessons-from-a-microservice-fault-classifier/6aff4baec27e45f5b21f553d)

Comment [Evaluation Isn't One Number: Lessons from a Microservice Fault Classifier](https://hashnode.com/posts/evaluation-isn-t-one-number-lessons-from-a-microservice-fault-classifier/6aff4baec27e45f5b21f553d)

[JustOneCX](https://hashnode.com/@justonecx)

[3h ago](https://hashnode.com/posts/evaluation-isn-t-one-number-lessons-from-a-microservice-fault-classifier/6aff4baec27e45f5b21f553d/comment/6b01dcb21412d11724da80e5)

The 0.80 versus 0.57 gap is the number that matters most. Did you hold out whole systems for the test set, or random windows from systems the model already saw?

[Jenna's Runbooks](https://hashnode.com/@jennasrunbooks)

[1h ago](https://hashnode.com/posts/evaluation-isn-t-one-number-lessons-from-a-microservice-fault-classifier/6aff4baec27e45f5b21f553d/comment/6b01dcb21412d11724da80e5#reply-6b01f53239579850f1a285c6)

Both, measured separately, and never with random windows. A random split would leak, because a case’s 5 windows are near-duplicates ( “The task and the data”). The 0.80 holds out whole cases from systems the model also trained on (“How the 0.80 was measured”), and the 0.57 holds out entire systems (“The unseen system”).
