Decoding Strategies in NLP Explained: From Greedy Search to Top-P Sampling (and How to Tune Them)

Introduction In LLM text generation, two steps are involved: (1) inference, where the model predicts raw scores called logits (raw score before turning it into a probability), The bigger the logit, the model thinks the token is more likely. But logit...