---
title: "Discussion on \"Tokenisation under the Hood\" | Hashnode"
description: "Discussion on \"Tokenisation under the Hood\". Tokenisation From Scratch: Building the Bridge Between Text and AI\nUnderstanding the translation layer that powers every large language model\nWhen you type a message to ChatGPT, Llama, or any other language model, something magical happens behind the..."
url: "https://hashnode.com/posts/tokenisation-under-the-hood/68f5f810f355a838b414c424"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Tokenisation under the Hood](https://llmtokenizer.hashnode.dev/tokenisation-under-the-hood?utm_source=hashnode&utm_medium=feed)

Tokenisation From Scratch: Building the Bridge Between Text and AI Understanding the translation layer that powers every large language model When you type a message to ChatGPT, Llama, or any other language model, something magical happens behind the...

Oct 20, 2025 · 10 min read · llmtokenizer.hashnode.dev  
Read the article

-   [#llm](https://hashnode.com/tag/llm)
-   [#tokenization](https://hashnode.com/tag/tokenization)
-   [#unicode](https://hashnode.com/tag/unicode)
-   [#tiktoken](https://hashnode.com/tag/tiktoken)
-   [#sentencepiece](https://hashnode.com/tag/sentencepiece)
-   [#llama](https://hashnode.com/tag/llama)
-   [#openai](https://hashnode.com/tag/openai)
-   [#gpt-4](https://hashnode.com/tag/gpt-4)
-   [#gpt-2](https://hashnode.com/tag/gpt-2)

## Written by

[Pornima Gaikwad](https://hashnode.com/@pornima1)

@pornima1

  
[View profile](https://hashnode.com/@pornima1)

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
