Putting the judge in the payment path and failing closed is the right call, and narrowing the prompt to per-claim grounding rather than overall quality is the key detail. One thing I'd be curious about: do you ever run the judge on a deliberately mismatched photo/description pair (e.g. a valid description attached to a photo of a different unit) as a canary? Since both models can share the same blind spots, a small rotating set of known-bad pairs is a cheap way to catch the judge drifting toward agreeing with the generator.
Yunus Emre Vurgun
Writing about AI agents, hardware, and practical software.
Putting the judge in the payment path and failing closed is the right call, and narrowing the prompt to per-claim grounding rather than overall quality is the key detail. One thing I'd be curious about: do you ever run the judge on a deliberately mismatched photo/description pair (e.g. a valid description attached to a photo of a different unit) as a canary? Since both models can share the same blind spots, a small rotating set of known-bad pairs is a cheap way to catch the judge drifting toward agreeing with the generator.