FeedDiscussion
The Ruler Can Change Too: Navigating Judge Drift in Production AI Evaluation
The Monday Morning Regression It is Monday morning. You arrive at your desk to find that production monitoring has flagged a sudden, severe degradation in your core Retrieval-Augmented Generation (RAG
nirizan.hashnode.dev8 min read
#ai-evaluation#llm-evaluation#llm-as-a-judge#rag-triad#judge-drift#ai-reliability#rag-evaluation#rag#retrieval-augmented-generation#ai-observability#ai-infrastructure#statistical-testing#machine-learning#nirizan#llm#artificial-intelligence
Judge drift is one of the most under-discussed failure modes in eval, so it is good to see it named directly. The trap I hit is that when the score drops you cannot tell if the system got worse or the judge did, unless you pin the judge model version and keep a small fixed set you re-score every release. I wrote a related piece on why I stopped scoring only the final answer for agent apps: kartiknvjk.hashnode.dev/evaluating-openai-agents-… . How are you separating judge drift from genuine model regression in production?