---
title: "Discussion on \"Understanding Tokenization\" | Hashnode"
description: "Discussion on \"Understanding Tokenization\". How AI Breaks Language Into Pieces It Can Predict"
url: "https://hashnode.com/posts/understanding-tokenization/69647b4351342afac0305cc7"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Understanding Tokenization](https://blog.devwithjay.com/understanding-tokenization?utm_source=hashnode&utm_medium=feed)

How AI Breaks Language Into Pieces It Can Predict

Jan 12 · 4 min read · blog.devwithjay.com  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#genai](https://hashnode.com/tag/genai)
-   [#tokenization](https://hashnode.com/tag/tokenization)
-   [#chaicode](https://hashnode.com/tag/chaicode)
-   [#llm](https://hashnode.com/tag/llm)

## Written by

[Jay Kadlag](https://hashnode.com/@devwithjaydotco)

@devwithjaydotco · Pune, India

  
[View profile](https://hashnode.com/@devwithjaydotco)

Ui » Ux » Web Dev

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
