SMSOUMIK Mahatoinlearnernew.hashnode.dev·Sep 17 · 5 min read "attention is all you need" - - Decoding Jargons.exe#.Tokenization its the process of converting text into a smaller pieces(token) which is then converted by a LLM to its familiar language which is numerical ID. a token can be : A complete word Part 01K
