---
title: "Discussion on \"Inside an LLM Tokenizer: Unicode, UTF-8, Bytes, and BPE\" | Hashnode"
description: "Discussion on \"Inside an LLM Tokenizer: Unicode, UTF-8, Bytes, and BPE\". Go to https://platform.openai.com/tokenizer\nPut 🏡 in the text box. And see the output.\n\n\n\n\nWhy one single emoji is giving us two tokenIDs?\nFor that, we need to understand three different things:\n\nUni"
url: "https://hashnode.com/posts/inside-an-llm-tokenizer-unicode-utf-8-bytes-and-bpe/6addf0ee96cee1a55fac01c7"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Inside an LLM Tokenizer: Unicode, UTF-8, Bytes, and BPE](https://rounakshow.hashnode.dev/inside-an-llm-tokenizer-unicode-utf-8-bytes-and-bpe?utm_source=hashnode&utm_medium=feed)

Go to https://platform.openai.com/tokenizer Put 🏡 in the text box. And see the output. Why one single emoji is giving us two tokenIDs? For that, we need to understand three different things: Uni

Oct 25 · 9 min read · rounakshow.hashnode.dev  
Read the article

-   [#llm](https://hashnode.com/tag/llm)
-   [#ai](https://hashnode.com/tag/ai)
-   [#tokenization](https://hashnode.com/tag/tokenization)

## Written by

[Rounak Show](https://hashnode.com/@ShowRounak)

@ShowRounak · Remote

  
[View profile](https://hashnode.com/@ShowRounak)

Learning Data Science

Learning Data Science and Sharing the journey through Hashnode.

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
