Tag feed

#bpe

4 posts0 followers

BPE vs WordPiece vs SentencePiece: A Beginner-Friendly Guide to Subword Tokenization

Aug 26, 2025 · 10 min read · Introduction Machines can’t directly understand words. It only know numbers. That’s why we need tokenization, a way to break text into smaller units (tokens) that can be mapped to numbers. There are three common levels of tokenization: word-level, ch...

Join discussion

RKRahul Kumardata-science-notes.hashnode.dev

0

🔍 Zero to Hero: Tokenization in NLP – From Basics to Subword Models

Apr 8, 2025 · 7 min read · ✨ “How can machines read language?” That's the fundamental question behind tokenization, the process of breaking down raw text into manageable, machine-readable pieces. If you're on your journey to becoming a data scientist or working with Natural La...

Join discussion

#bpe

Search Hashnode

#bpe

BPE vs WordPiece vs SentencePiece: A Beginner-Friendly Guide to Subword Tokenization

🔍 Zero to Hero: Tokenization in NLP – From Basics to Subword Models