---
title: "Discussion on \"How to Train BPE, WordPiece, and Unigram Tokenizers from Scratch using Hugging Face\" | Hashnode"
description: "Discussion on \"How to Train BPE, WordPiece, and Unigram Tokenizers from Scratch using Hugging Face\". If you've had some experience with NLP, you probably know that tokenization is at the helm of any NLP pipeline.\nTokenization is often regarded as a subfield of NLP but it has its own story of evolution. And now it underpins many state-of-the-art NLP ..."
url: "https://hashnode.com/posts/train-algorithms-from-scratch-with-hugging-face/66d45f5ab3016bf139028d4c"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [How to Train BPE, WordPiece, and Unigram Tokenizers from Scratch using Hugging Face](https://freecodecamp.org/news/train-algorithms-from-scratch-with-hugging-face?utm_source=hashnode&utm_medium=feed)

If you've had some experience with NLP, you probably know that tokenization is at the helm of any NLP pipeline. Tokenization is often regarded as a subfield of NLP but it has its own story of evolution. And now it underpins many state-of-the-art NLP ...

Oct 18, 2021 · 10 min read · freecodecamp.org  
Updated Sep 26, 2024  
Read the article

-   [#algorithms](https://hashnode.com/tag/algorithms)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#natural-language-processing](https://hashnode.com/tag/natural-language-processing)
-   [#nlp](https://hashnode.com/tag/nlp)

## Written by

[Harshit Tyagi](https://hashnode.com/@dswh)

@dswh

  
[View profile](https://hashnode.com/@dswh)

Founder, Ex-Director Data Science & ML

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
