---
title: "Comment by Rishita Sharma on \"How to Set Up Evals for a Complex Multi-Agent System\" | Hashnode"
description: "The bit about a coordinator following a specialist's wrong advice is the failure I hit most. Each agent looked fine when I tested it alone, and the final answer"
url: "https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac5cd1c75fe3e50fe00c0ba"
---

[Discussion](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03)

Comment [How to Set Up Evals for a Complex Multi-Agent System](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03)

[Rishita Sharma](https://hashnode.com/@narawaybuilder)

[Oct 7](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac5cd1c75fe3e50fe00c0ba)

The bit about a coordinator following a specialist's wrong advice is the failure I hit most. Each agent looked fine when I tested it alone, and the final answer was still wrong because of what got passed between them. Grading the whole episode instead of the last message is what finally showed me where it broke. How many reviewed cases did you need before the judge felt calibrated enough to trust?

[Gary Stafford](https://hashnode.com/@garystafford)

[2d ago](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac5cd1c75fe3e50fe00c0ba#reply-6af3690098414658a8198689)

Thanks, Rishita. We used 27 saved responses for the calibration pilot. The judge agreed with 26 of the 27 original labels. Those examples came from development runs and the labels were reviewed with Codex, so I treated that agreement as evidence about how the rubric behaved on that set.

For a decision that needs more confidence, I’d freeze the judge and rubric, then check them on fresh examples labeled by domain reviewers. I’d look especially closely at false passes on consequential mistakes, like a coordinator dropping a caveat that should have changed the plan.