Explain Tokenization to fresher

What is Tokenization? Tokenization is the process of converting text into smaller units called tokens. These tokens can be: Words — e.g., "I love pizza" → ["I", "love", "pizza"] Sub words — e.g., "ChatGPT" → ["Chat", "G", "PT"] Characters — e.g., ...