--- title: "#byte-pair-encoding articles | Hashnode" description: "Explore 8 articles tagged #byte-pair-encoding on Hashnode. Tutorials, guides, and how-tos from the community." url: "https://hashnode.com/tag/byte-pair-encoding" --- Tag # #byte-pair-encoding 8 posts · 0 followers [Rahul Sai Indeevar V](https://hashnode.com/@rahul-ai) · [Sep 2](https://hashnode.com/posts/demystifying-tokenization-from-bpe-and-wordpiece-to-sentencepiece/6a986df8c4508e37669f4e40) ### [Demystifying Tokenization: From BPE and WordPiece to SentencePiece](https://rahul-ai.hashnode.dev/demystifying-tokenization-from-bpe-and-wordpiece-to-sentencepiece?utm_source=hashnode&utm_medium=feed) Tokenization is the critical bridge between raw human language and numerical vector spaces in modern Large Language Models (LLMs). Before a Transformer ever sees a sentence, a tokenizer breaks down ra 12 min read · [rahul-ai.hashnode.dev](https://rahul-ai.hashnode.dev/demystifying-tokenization-from-bpe-and-wordpiece-to-sentencepiece?utm_source=hashnode&utm_medium=feed) [PUNITH](https://hashnode.com/@punith-v) · [Jun 28](https://hashnode.com/posts/how-i-built-a-bpe-tokenizer-from-scratch-part-1-training/6a416a2f3abbac56183b9a5d) ### [How I Built a BPE Tokenizer from Scratch (Part 1: Training)](https://bpe-tokenizer.hashnode.dev/how-i-built-a-bpe-tokenizer-from-scratch-part-1-training?utm_source=hashnode&utm_medium=feed) I wanted to actually understand how tokenization works — not just use a library. So I built a BPE tokenizer from scratch. This is Part 1: training the tokenizer, counting pairs, and building the vocabulary. 5 min read · [bpe-tokenizer.hashnode.dev](https://bpe-tokenizer.hashnode.dev/how-i-built-a-bpe-tokenizer-from-scratch-part-1-training?utm_source=hashnode&utm_medium=feed) [Tushar Singla](https://hashnode.com/@tusharsingla) · [Mar 24](https://hashnode.com/posts/i-built-the-algorithm-behind-chatgpt-from-scratch-here-s-what-i-learned/69c23eab30a9b81e3a1788b9) ### [I built the algorithm behind ChatGPT from scratch — here's what I learned](https://queued.hashnode.dev/i-built-the-algorithm-behind-chatgpt-from-scratch-here-s-what-i-learned?utm_source=hashnode&utm_medium=feed) How I built a bilingual BPE tokenizer in pure Python — no ML libraries, trained on 1GB of English + Hindi text. Here's what I learned. 11 min read · [queued.hashnode.dev](https://queued.hashnode.dev/i-built-the-algorithm-behind-chatgpt-from-scratch-here-s-what-i-learned?utm_source=hashnode&utm_medium=feed) [Harshavardhanan](https://hashnode.com/@harshavardhan) · [Feb 3](https://hashnode.com/posts/how-bpe-tokenization-works-llm-algorithm/69823ebfd9f0256419abf8a6) ### [How Tokenization Works: BPE and the Algorithm Behind Your LLM](https://blog.pragmaticbyharsh.com/how-bpe-tokenization-works-llm-algorithm?utm_source=hashnode&utm_medium=feed) Learn how Byte Pair Encoding (BPE) actually works — the algorithm that powers GPT, Claude, and LLaMA tokenizers. Step-by-step with examples. 9 min read · [blog.pragmaticbyharsh.com](https://blog.pragmaticbyharsh.com/how-bpe-tokenization-works-llm-algorithm?utm_source=hashnode&utm_medium=feed) [Mohamad Mahmood](https://hashnode.com/@mohamadrazzimy) · [Feb 2](https://hashnode.com/posts/byte-pair-encoding-for-text-tokenization-in-natural-language-processing/69801327b15f3aaa47c85bda) ### [Byte-Pair Encoding for Text Tokenization in Natural Language Processing](https://textlab.razzi.my/byte-pair-encoding-for-text-tokenization-in-natural-language-processing?utm_source=hashnode&utm_medium=feed) 1\. Introduction Modern natural language processing (NLP) systems do not operate directly on raw text. Instead, text must first be transformed into a structured representation that models can process efficiently. One of the most important steps in thi... 5 min read · [textlab.razzi.my](https://textlab.razzi.my/byte-pair-encoding-for-text-tokenization-in-natural-language-processing?utm_source=hashnode&utm_medium=feed) [Apurva kanth](https://hashnode.com/@Apurvak3) · [Oct 29, 2025](https://hashnode.com/posts/neural-machine-translation-of-rare-words-with-subword-units/69020d21300d11ed9864ad56) ### [\# 🧠 Neural Machine Translation of Rare Words with Subword Units —](https://apurvak3.hashnode.dev/neural-machine-translation-of-rare-words-with-subword-units?utm_source=hashnode&utm_medium=feed) "How breaking words into subword units (like BPE) revolutionized translation and NLP" 4 min read · [apurvak3.hashnode.dev](https://apurvak3.hashnode.dev/neural-machine-translation-of-rare-words-with-subword-units?utm_source=hashnode&utm_medium=feed) [Supriya Kadam Daberao](https://hashnode.com/@supriya-kd) · [Sep 15, 2025](https://hashnode.com/posts/tokenization-for-freshers-with-pizza/68c7ee6a2b37dce5330970ba) ### [Explaining Tokenization to Freshers: From Pizza Slices 🍕 to Data 💻🧠✨](https://www.ruhmani.com/tokenization-for-freshers-with-pizza?utm_source=hashnode&utm_medium=feed) Explain Tokenization to Freshers. Learn the basics of tokenization as we transform sentences into numbers through Byte-Pair Encoding. 17 min read · [ruhmani.com](https://www.ruhmani.com/tokenization-for-freshers-with-pizza?utm_source=hashnode&utm_medium=feed) [Girish Kumar Adari](https://hashnode.com/@GirishKumarAdari) · [Sep 16, 2023](https://hashnode.com/posts/unraveling-the-byte-pair-encoding-bpe-algorithm-in-nlp/650619b6ca70366e0deb028f) ### [Unraveling the Byte Pair Encoding (BPE) Algorithm in NLP](https://girishkumaradari.hashnode.dev/unraveling-the-byte-pair-encoding-bpe-algorithm-in-nlp?utm_source=hashnode&utm_medium=feed) Introduction As data continues to proliferate in volume and complexity, the need for effective methods to understand and analyze it becomes more pressing. This is especially true in the world of Natural Language Processing (NLP), where text, a unique... 6 min read · [girishkumaradari.hashnode.dev](https://girishkumaradari.hashnode.dev/unraveling-the-byte-pair-encoding-bpe-algorithm-in-nlp?utm_source=hashnode&utm_medium=feed)