---
title: "Discussion on \"How I Built an LLM-as-a-Judge Eval System for a Medical Voice Agent (First 14 Scores)\" | Hashnode"
description: "Discussion on \"How I Built an LLM-as-a-Judge Eval System for a Medical Voice Agent (First 14 Scores)\". How I built an LLM-as-a-judge eval system for Aiko, a medical voice agent: ten cases, 14 graded visits and where the judge went wrong."
url: "https://hashnode.com/posts/llm-judge-eval-system-medical-voice-agent/6af3ba8a78ba7b79704a7601"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [How I Built an LLM-as-a-Judge Eval System for a Medical Voice Agent (First 14 Scores)](https://aiko-realtime-eval.hashnode.dev/llm-judge-eval-system-medical-voice-agent?utm_source=hashnode&utm_medium=feed)

How I built an LLM-as-a-judge eval system for Aiko, a medical voice agent: ten cases, 14 graded visits and where the judge went wrong.

13h ago · 18 min read · aiko-realtime-eval.hashnode.dev  
Read the article

-   [#llm](https://hashnode.com/tag/llm)
-   [#eval](https://hashnode.com/tag/eval)
-   [#healthcare](https://hashnode.com/tag/healthcare)
-   [#voice-ai](https://hashnode.com/tag/voice-ai)
-   [#ai-agents](https://hashnode.com/tag/ai-agents)
-   [#ai](https://hashnode.com/tag/ai)

## Written by

[Harsh Shelke](https://hashnode.com/@harsh180502)

@harsh180502 · Seattle, USA

  
[View profile](https://hashnode.com/@harsh180502)

ML Research Engineer at Gmango AI Health

I work on a live voice agentic system at Gmango, an early stage med-tech startup in Bellevue,WA,USA . I write about evals, develop voice agentic systems end to end, and the parts that only break once a call is live.

## Responses 2

Markdown supported · ⌘↵ to post

[OneLifeVibe](https://hashnode.com/@onelifevibe)

[5h ago](https://hashnode.com/posts/llm-judge-eval-system-medical-voice-agent/6af3ba8a78ba7b79704a7601/comment/6af430fe813891c5832760ad)

The decision that stood out to me is keeping the critical checks outside the average. A visit scoring 0.9 that misses a safety check should not read as a near-pass, and the failed-critical flag stops that averaging-away problem in a way a single 0.63 average never could. The split between critical at 82% hit rate and must-ask at 53% tells a much clearer story than the headline score alone.

I also liked that the judge never does the arithmetic. Having GPT-4o mini return a verdict and a quote per check, then calculating the weighted score in plain code, keeps the number auditable, and the clickable score card that scrolls to the quoted Aiko line makes that audit practical. The "does not" finding is a useful warning too: rewording those checks into positive actions the judge can actually find in the transcript seems like the right fix, rather than trusting a model to reliably confirm an absence.

[AI Developer](https://hashnode.com/@lycore)

[11h ago](https://hashnode.com/posts/llm-judge-eval-system-medical-voice-agent/6af3ba8a78ba7b79704a7601/comment/6af3dcdfb525370e2ad74f41)

Separating critical misses from the weighted average is important, especially when the same case produced similar scores but different safety outcomes. Since the judge sometimes cited the patient rather than the agent, a speaker-aware evidence check plus clinician review of a blinded sample could estimate false passes as well as false failures. Repeating cases with the same rubric and reporting per-case variation would also make comparisons less dependent on a single simulated conversation.