Tag feed

#byte-pair-encoding

7 posts0 followers

Explore Hashnode

Alternatives

PPUNITHbpe-tokenizer.hashnode.devJun 28 · 5 min read

How I Built a BPE Tokenizer from Scratch (Part 1: Training)

Okay so before we get into the code, let me tell you why I even did this. I could've just used a library. tiktoken exists. HuggingFace tokenizers exist. But I wanted to actually get it — like really u

0

TSTushar Singlaqueued.hashnode.devMar 24 · 11 min read

I built the algorithm behind ChatGPT from scratch — here's what I learned

"What I cannot create, I do not understand." — Richard Feynman That quote hit different when I was staring at my screen at 2am, watching my tokenizer learn the word "the" by merge #17. The origin st

0

HHarshavardhananblog.pragmaticbyharsh.comFeb 3 · 9 min read

How Tokenization Works: BPE and the Algorithm Behind Your LLM

Every time you send a message to GPT-4 or Claude, an algorithm from 1994 decides how much you'll pay. That algorithm is Byte Pair Encoding — BPE for short. It's not glamorous, but it's running under the hood of nearly every modern LLM. Once you under...

0

MMMohamad Mahmoodtextlab.razzi.myFeb 2 · 5 min read

Byte-Pair Encoding for Text Tokenization in Natural Language Processing

1. Introduction Modern natural language processing (NLP) systems do not operate directly on raw text. Instead, text must first be transformed into a structured representation that models can process efficiently. One of the most important steps in thi...

0

AKApurva kanthapurvak3.hashnode.devOct 29, 2025 · 4 min read

# 🧠 Neural Machine Translation of Rare Words with Subword Units —

By Apurva KanthPublished on Hashnode 💭 The Problem: When Neural Networks Don’t Know a Word Traditional Neural Machine Translation (NMT) systems work with a fixed vocabulary — often limited to around 30,000–50,000 words. But languages are full of ra...

0

SKSupriya Kadam Daberaoruhmani.comSep 15, 2025 · 17 min read

Explaining Tokenization to Freshers: From Pizza Slices 🍕 to Data 💻🧠✨

🎉 Welcome, brave fresher! ⚔️ to the fascinating and occasionally bewildering world of Machine Learning! You’ve probably heard the mantra: “Computers don’t understand words; they understand numbers.” It’s the fundamental law of the land, the secret s...

0

GKGirish Kumar Adarigirishkumaradari.hashnode.devSep 16, 2023 · 6 min read

Unraveling the Byte Pair Encoding (BPE) Algorithm in NLP

Introduction As data continues to proliferate in volume and complexity, the need for effective methods to understand and analyze it becomes more pressing. This is especially true in the world of Natural Language Processing (NLP), where text, a unique...

0

#byte-pair-encoding

Search Hashnode

#byte-pair-encoding

Explore Hashnode

How I Built a BPE Tokenizer from Scratch (Part 1: Training)

I built the algorithm behind ChatGPT from scratch — here's what I learned

How Tokenization Works: BPE and the Algorithm Behind Your LLM

Byte-Pair Encoding for Text Tokenization in Natural Language Processing

# 🧠 Neural Machine Translation of Rare Words with Subword Units —

Explaining Tokenization to Freshers: From Pizza Slices 🍕 to Data 💻🧠✨

Unraveling the Byte Pair Encoding (BPE) Algorithm in NLP

Trending tags this week