Tokenization - For Freshers

Imagine you have a big storybook .If you try to read the whole thing in one breath, you’ll pass out faster than India chasing 350 without Kohli.So you split it into words and sentences. Now you can read it bit by bit.That’s exactly what tokenization ...