Temperature, Top-k, and Top-p: How Decoding Changes What a GPT Model Says

TL;DR A GPT model does not directly choose its next word. More precisely, it assigns probabilities to thousands of possible next tokens. Decoding is the step that converts those probabilities into an