"attention is all you need" - - Decoding Jargons.exe
#.Tokenization
its the process of converting text into a smaller pieces(token) which is then converted by a LLM to its familiar language which is numerical ID. a token can be :
A complete word
Part
