---
title: "Discussion on \"Tokenizers in NLP: Word, Character, and Sub-Word Models\" | Hashnode"
description: "Discussion on \"Tokenizers in NLP: Word, Character, and Sub-Word Models\". In the world of Natural Language Processing (NLP), the first step in almost every pipeline is tokenization — breaking raw text into smaller units, known as tokens. These tokens serve as the building blocks that machine learning models, especially Lar..."
url: "https://hashnode.com/posts/tokenizers-in-nlp-word-character-and-sub-word-models/68cb9363e4711bc32e4ed526"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Tokenizers in NLP: Word, Character, and Sub-Word Models](https://tokenizermodels.hashnode.dev/tokenizers-in-nlp-word-character-and-sub-word-models?utm_source=hashnode&utm_medium=feed)

In the world of Natural Language Processing (NLP), the first step in almost every pipeline is tokenization — breaking raw text into smaller units, known as tokens. These tokens serve as the building blocks that machine learning models, especially Lar...

Sep 18, 2025 · 3 min read · tokenizermodels.hashnode.dev  
Read the article

-   [#nlp-transformers](https://hashnode.com/tag/nlp-transformers)
-   [#tokenization](https://hashnode.com/tag/tokenization)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#deep-learning](https://hashnode.com/tag/deep-learning)
-   [#techblog](https://hashnode.com/tag/techblog)
-   [#ai](https://hashnode.com/tag/ai)
-   [#bert](https://hashnode.com/tag/bert)
-   [#llm](https://hashnode.com/tag/llm)
-   [#transformers](https://hashnode.com/tag/transformers)
-   [#gpt](https://hashnode.com/tag/gpt)

## Written by

[Sumanth Mandavalli](https://hashnode.com/@SumanthMandavalli)

@SumanthMandavalli · Hyderabad

  
[View profile](https://hashnode.com/@SumanthMandavalli)

Developer

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
