The bit about a coordinator following a specialist's wrong advice is the failure I hit most. Each agent looked fine when I tested it alone, and the final answer was still wrong because of what got passed between them. Grading the whole episode instead of the last message is what finally showed me where it broke. How many reviewed cases did you need before the judge felt calibrated enough to trust?
Good walkthrough. The point about grading answer quality and workflow compliance separately stood out — it's easy to trust a pass rate without noticing the agent took an expensive or non-compliant path to get there. The section on deliberately failing runs is useful too; seeing how an eval breaks is the fastest way to trust it.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Handoffs are where I lose the most time, especially the 'lost uncertainty' case you name: a specialist returns a hedged, evidence-light recommendation and the coordinator reads it as settled fact. Freezing the specialist reports so rival coordinators get identical input is the step I would have skipped. When you grade a handoff, do you assert on the preserved caveats directly, or only on whether the final plan stayed within authority?