---
title: "Discussion on \"How to Set Up Evals for a Complex Multi-Agent System\" | Hashnode"
description: "Discussion on \"How to Set Up Evals for a Complex Multi-Agent System\". Using Codex and Promptfoo to define success, diagnose failures, and choose models"
url: "https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [How to Set Up Evals for a Complex Multi-Agent System](https://blog.latentthoughts.com/how-to-set-up-evals-for-a-complex-multi-agent-system?utm_source=hashnode&utm_medium=feed)

Using Codex and Promptfoo to define success, diagnose failures, and choose models

Oct 6 · 29 min read · blog.latentthoughts.com  
Updated Oct 9  
Read the article

-   [#promptfoo](https://hashnode.com/tag/promptfoo)
-   [#ai-agents](https://hashnode.com/tag/ai-agents)
-   [#multi-agent-systems](https://hashnode.com/tag/multi-agent-systems)
-   [#model-evaluation](https://hashnode.com/tag/model-evaluation)
-   [#codex](https://hashnode.com/tag/codex)
-   [#openai](https://hashnode.com/tag/openai)

## Written by

[Gary Stafford](https://hashnode.com/@garystafford)

@garystafford · New York City, USA

  
[View profile](https://hashnode.com/@garystafford)

Practical AI engineering: models, agents, and evaluations: from experiments to real-world applications.

Applied AI Engineer at OpenAI. I write about building AI applications, evaluating models, and turning experiments into practical solutions, drawing on my background in cloud architecture and software engineering.

## Responses 5

Markdown supported · ⌘↵ to post

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Oct 7](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac6966a95ff0ef484394708)

Handoffs are where I lose the most time, especially the 'lost uncertainty' case you name: a specialist returns a hedged, evidence-light recommendation and the coordinator reads it as settled fact. Freezing the specialist reports so rival coordinators get identical input is the step I would have skipped. When you grade a handoff, do you assert on the preserved caveats directly, or only on whether the final plan stayed within authority?

[Gary Stafford](https://hashnode.com/@garystafford)

[Nov 10](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac6966a95ff0ef484394708#reply-6af368f36f6fdf3b91b9a6b1)

Thanks, Kartik. We checked both. In the Mars replay, each Director configuration had access to the same recorded specialist reports. The deterministic checks verified that the Director proposed an allowed set of actions and submitted it for approval. The semantic rubric separately checked whether the Director distinguished projections from observations and disclosed material shortfalls.

For a dedicated handoff test, I’d make the caveat explicit in the fixture. Say a specialist reports normal air-processing throughput but says cabin air quality is unmeasured. I’d check that the Director carries that uncertainty into the plan or resolves it with new evidence. A plan can stay within its authority and still overstate what the team knows.

[Rishita Sharma](https://hashnode.com/@narawaybuilder)

[Oct 7](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac5cd1c75fe3e50fe00c0ba)

The bit about a coordinator following a specialist's wrong advice is the failure I hit most. Each agent looked fine when I tested it alone, and the final answer was still wrong because of what got passed between them. Grading the whole episode instead of the last message is what finally showed me where it broke. How many reviewed cases did you need before the judge felt calibrated enough to trust?

[Gary Stafford](https://hashnode.com/@garystafford)

[Nov 10](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac5cd1c75fe3e50fe00c0ba#reply-6af3690098414658a8198689)

Thanks, Rishita. We used 27 saved responses for the calibration pilot. The judge agreed with 26 of the 27 original labels. Those examples came from development runs and the labels were reviewed with Codex, so I treated that agreement as evidence about how the rubric behaved on that set.

For a decision that needs more confidence, I’d freeze the judge and rubric, then check them on fresh examples labeled by domain reviewers. I’d look especially closely at false passes on consequential mistakes, like a coordinator dropping a caveat that should have changed the plan.

[netproxy](https://hashnode.com/@argumentmoney8117)

[Oct 6](https://hashnode.com/posts/how-to-set-up-evals-for-a-complex-multi-agent-system/6ac46769848e3306fd9c5c03/comment/6ac57556d736df1d66b7e3e6)

Good walkthrough. The point about grading answer quality and workflow compliance separately stood out — it's easy to trust a pass rate without noticing the agent took an expensive or non-compliant path to get there. The section on deliberately failing runs is useful too; seeing how an eval breaks is the fastest way to trust it.
