---
title: "#tokenization-in-ai articles | Hashnode"
description: "Explore 6 articles tagged #tokenization-in-ai on Hashnode. Tutorials, guides, and how-tos from the community."
url: "https://hashnode.com/tag/tokenization-in-ai"
---

Tag

# #tokenization-in-ai

6 posts · 0 followers

[Mohitm](https://hashnode.com/@mohitm) · [Oct 28](https://hashnode.com/posts/why-your-llm-can-t-count-the-r-s-in-strawberry/6ae15bd87700189050d03c0f)

### [Why Your LLM Can't Count the R's in 'Strawberry'](https://monthlydatascience.hashnode.dev/why-your-llm-can-t-count-the-r-s-in-strawberry?utm_source=hashnode&utm_medium=feed)

Ask ChatGPT how many R's are in strawberry and it often gets it wrong. The reason isn't a bug — it's tokenization. Here's what's actually happening, with an interactive demo.

6 min read · [monthlydatascience.hashnode.dev](https://monthlydatascience.hashnode.dev/why-your-llm-can-t-count-the-r-s-in-strawberry?utm_source=hashnode&utm_medium=feed)

[1](https://hashnode.com/posts/why-your-llm-can-t-count-the-r-s-in-strawberry/6ae15bd87700189050d03c0f)

[Viru Jade](https://hashnode.com/@viru09) · [May 15](https://hashnode.com/posts/i-built-a-tokenizer-from-scratch-here-is-what-i-learned/6a06dd210111c739e499264f)

### [I Built a Tokenizer From Scratch. Here Is What I Learned.](https://virujade.hashnode.dev/i-built-a-tokenizer-from-scratch-here-is-what-i-learned?utm_source=hashnode&utm_medium=feed)

And why GPT-4o does it completely differently.

8 min read · [virujade.hashnode.dev](https://virujade.hashnode.dev/i-built-a-tokenizer-from-scratch-here-is-what-i-learned?utm_source=hashnode&utm_medium=feed)

[Arun Chauhan](https://hashnode.com/@arunchauhan) · [Aug 20, 2025](https://hashnode.com/posts/tokenization-in-ai/68a5ad73737b544628ba2b0c)

### [Tokenization in AI](https://arunchauhan-gen-ai.hashnode.dev/tokenization-in-ai?utm_source=hashnode&utm_medium=feed)

The First Step in Teaching Machines to Understand Language

4 min read · [arunchauhan-gen-ai.hashnode.dev](https://arunchauhan-gen-ai.hashnode.dev/tokenization-in-ai?utm_source=hashnode&utm_medium=feed)

[Naveen Kumar](https://hashnode.com/@soynerd) · [Aug 13, 2025](https://hashnode.com/posts/ai-tokenization/689c9502de05741eb5964319)

### [Tokenization](https://naveenweb.hashnode.dev/ai-tokenization?utm_source=hashnode&utm_medium=feed)

Think of tokenization like chopping up a sentence into Lego blocks, so the AI can build meaning block-by-block. The process of breaking text into smaller pieces, called tokens, so a language model ( like chat-GPT , Gemini ) can understand and work wi...

2 min read · [naveenweb.hashnode.dev](https://naveenweb.hashnode.dev/ai-tokenization?utm_source=hashnode&utm_medium=feed)
