Don’t A/B Test the LLM: Randomize the Voice Conversation Policy

A new LLM can sound clearly better in a demo and still make a voice companion worse in production. The tension is not simply model quality versus model cost. In a real-time conversation, changing the