Inside an LLM Tokenizer: Unicode, UTF-8, Bytes, and BPE

Go to https://platform.openai.com/tokenizer Put 🏡 in the text box. And see the output. Why one single emoji is giving us two tokenIDs? For that, we need to understand three different things: Uni