Why Do Chatbots Fall Into The Hallucination Trap: How Chatbots Learn to Guess Instead of Admit 'I Don’t Know'?

Chatbots sometimes provide answers that sound confident even when they are wrong. This phenomenon, known as the hallucination trap, occurs when models choose a guess over admitting uncertainty. In this post, we explain why this happens and what can b...