---
title: "#tokenization articles | Hashnode"
description: "Explore 449 articles tagged #tokenization on Hashnode. Tutorials, guides, and how-tos from the community."
url: "https://hashnode.com/tag/tokenization"
---

Tag

# #tokenization

449 posts · 9 followers

[DataNorthAI](https://hashnode.com/@datanorthai) · [Nov 3](https://hashnode.com/posts/how-tokenization-works-in-llms/6ae99f69f250122bafb87743)

### [How tokenization works in LLMs](https://datanorth.hashnode.dev/how-tokenization-works-in-llms?utm_source=hashnode&utm_medium=feed)

Learn how LLM tokenization works, compare word, character and subword methods, and understand how tokens affect API costs and model performance.

4 min read · [datanorth.hashnode.dev](https://datanorth.hashnode.dev/how-tokenization-works-in-llms?utm_source=hashnode&utm_medium=feed)

[Mohitm](https://hashnode.com/@mohitm) · [Oct 28](https://hashnode.com/posts/why-your-llm-can-t-count-the-r-s-in-strawberry/6ae15bd87700189050d03c0f)

### [Why Your LLM Can't Count the R's in 'Strawberry'](https://monthlydatascience.hashnode.dev/why-your-llm-can-t-count-the-r-s-in-strawberry?utm_source=hashnode&utm_medium=feed)

Ask ChatGPT how many R's are in strawberry and it often gets it wrong. The reason isn't a bug — it's tokenization. Here's what's actually happening, with an interactive demo.

6 min read · [monthlydatascience.hashnode.dev](https://monthlydatascience.hashnode.dev/why-your-llm-can-t-count-the-r-s-in-strawberry?utm_source=hashnode&utm_medium=feed)

[1](https://hashnode.com/posts/why-your-llm-can-t-count-the-r-s-in-strawberry/6ae15bd87700189050d03c0f)

[Rounak Show](https://hashnode.com/@ShowRounak) · [Oct 25](https://hashnode.com/posts/inside-an-llm-tokenizer-unicode-utf-8-bytes-and-bpe/6addf0ee96cee1a55fac01c7)

### [Inside an LLM Tokenizer: Unicode, UTF-8, Bytes, and BPE](https://rounakshow.hashnode.dev/inside-an-llm-tokenizer-unicode-utf-8-bytes-and-bpe?utm_source=hashnode&utm_medium=feed)

Go to https://platform.openai.com/tokenizer Put 🏡 in the text box. And see the output. Why one single emoji is giving us two tokenIDs? For that, we need to understand three different things: Uni

9 min read · [rounakshow.hashnode.dev](https://rounakshow.hashnode.dev/inside-an-llm-tokenizer-unicode-utf-8-bytes-and-bpe?utm_source=hashnode&utm_medium=feed)

[Khasky](https://hashnode.com/@khasky) · [Oct 5](https://hashnode.com/posts/non-english-prompts-cost-more-tokens-context-and-money-in-most-llms/6ac3290df7e50888e2792a2a)

### [Non-English Prompts Cost More Tokens, Context and Money in Most LLMs](https://khasky.hashnode.dev/non-english-prompts-cost-more-tokens-context-and-money-in-most-llms?utm_source=hashnode&utm_medium=feed)

Claude Opus 5 costs $5 per million input tokens whether a prompt arrives in English or in Russian. What changes between the two languages is how many tokens the same paragraph becomes, and the bill is

7 min read · [khasky.hashnode.dev](https://khasky.hashnode.dev/non-english-prompts-cost-more-tokens-context-and-money-in-most-llms?utm_source=hashnode&utm_medium=feed)

[2](https://hashnode.com/posts/non-english-prompts-cost-more-tokens-context-and-money-in-most-llms/6ac3290df7e50888e2792a2a)

[Nagappan S](https://hashnode.com/@imkarthik03) · [Sep 24](https://hashnode.com/posts/tokens-tokenization-why-ai-counts-words-weirdly/6ab4cf96ce67b7fbc2815d87)

### [Tokens & Tokenization: Why AI "Counts" Words Weirdly](https://nagspidey.hashnode.dev/tokens-tokenization-why-ai-counts-words-weirdly?utm_source=hashnode&utm_medium=feed)

Tokens & Tokenization: Why AI Counts Words Weirdly

8 min read · [nagspidey.hashnode.dev](https://nagspidey.hashnode.dev/tokens-tokenization-why-ai-counts-words-weirdly?utm_source=hashnode&utm_medium=feed)

[1](https://hashnode.com/posts/tokens-tokenization-why-ai-counts-words-weirdly/6ab4cf96ce67b7fbc2815d87)

[Manas Shinde](https://hashnode.com/@manas-shinde) · [Sep 23](https://hashnode.com/posts/tokenization-from-first-principles-to-production-systems/6ab3d4f0cc23bb0e663fbc07)

### [Tokenization: From First Principles to Production Systems](https://manas-shindes-blogs.hashnode.dev/tokenization-from-first-principles-to-production-systems?utm_source=hashnode&utm_medium=feed)

Why Tokenization Matters

8 min read · [manas-shindes-blogs.hashnode.dev](https://manas-shindes-blogs.hashnode.dev/tokenization-from-first-principles-to-production-systems?utm_source=hashnode&utm_medium=feed)

[Arshad](https://hashnode.com/@arshadakl) · [Sep 21](https://hashnode.com/posts/same-business-same-ai-why-could-the-kerala-version-cost-more-to-run/6ab17616fea8d190ee6fa8a5)

### [Same Business. Same AI. Why Could the Kerala Version Cost More to Run?](https://blog.arshadakl.in/same-business-same-ai-why-could-the-kerala-version-cost-more-to-run?utm_source=hashnode&utm_medium=feed)

Why Malayalam AI can use more tokens than English and what that means for the cost of running AI applications in Kerala.

4 min read · [blog.arshadakl.in](https://blog.arshadakl.in/same-business-same-ai-why-could-the-kerala-version-cost-more-to-run?utm_source=hashnode&utm_medium=feed)

[Sunney Sood](https://hashnode.com/@sunnsood) · [Sep 18](https://hashnode.com/posts/tokens-tokenizers-how-a-computer-reads-a-sentence/6aad91396772d29a6b9e77ca)

### [Tokens & Tokenizers: How a Computer "Reads" a Sentence](https://sunneysood.hashnode.dev/tokens-tokenizers-how-a-computer-reads-a-sentence?utm_source=hashnode&utm_medium=feed)

Here's a weird fact: when you read the sentence "I love pizza", your brain doesn't see letters one by one — I, then space, then l, o, v, e... You just see the words. Your brain has already chopped the

17 min read · [sunneysood.hashnode.dev](https://sunneysood.hashnode.dev/tokens-tokenizers-how-a-computer-reads-a-sentence?utm_source=hashnode&utm_medium=feed)

[Rahul Sai Indeevar V](https://hashnode.com/@rahul-ai) · [Sep 2](https://hashnode.com/posts/demystifying-tokenization-from-bpe-and-wordpiece-to-sentencepiece/6a986df8c4508e37669f4e40)

### [Demystifying Tokenization: From BPE and WordPiece to SentencePiece](https://rahul-ai.hashnode.dev/demystifying-tokenization-from-bpe-and-wordpiece-to-sentencepiece?utm_source=hashnode&utm_medium=feed)

Tokenization is the critical bridge between raw human language and numerical vector spaces in modern Large Language Models (LLMs). Before a Transformer ever sees a sentence, a tokenizer breaks down ra

12 min read · [rahul-ai.hashnode.dev](https://rahul-ai.hashnode.dev/demystifying-tokenization-from-bpe-and-wordpiece-to-sentencepiece?utm_source=hashnode&utm_medium=feed)

[Internals Decoded](https://hashnode.com/@internalsdecoded) · [Aug 19](https://hashnode.com/posts/tokens-the-currency-of-ai/6a85b8a60ec76ee928628daa)

### [Tokens: The Currency of AI](https://internals-decoded.hashnode.dev/tokens-the-currency-of-ai?utm_source=hashnode&utm_medium=feed)

Why models see 'strawberry' as three pieces, why pricing is per token, and why that matters to you.

13 min read · [internals-decoded.hashnode.dev](https://internals-decoded.hashnode.dev/tokens-the-currency-of-ai?utm_source=hashnode&utm_medium=feed)
