---
title: "Discussion on \"What Are Tokens — And Why Should You Care?\" | Hashnode"
description: "Discussion on \"What Are Tokens — And Why Should You Care?\". The hidden unit of measurement that shapes every conversation you have with Claude.\n\nYou type a message to Claude. You hit send. A response flows back in seconds. Simple, right? But beneath that seamless exchange, something interesting is happening — your words are being sliced into tiny linguistic units called tokens before Claude ever \"reads\" them.\n\nTokens are the atomic unit of language for large language models. They're not characters, and they're not always full words. They sit somewhere in between — and understanding them unlocks a clearer picture of how AI language models actually work, why they have limits, and how to work with those limits instead of against them.\n\nSo what exactly is a token?\n\nThink of tokenisation as breaking text into the most useful chunks for a model to learn from. Common words like \"the\" or \"and\" are usually one token. Longer or rarer words might get split into two or three pieces. Punctuation, spaces, and newlines all count too.\n\nExample — how this sentence gets tokenised\nClaude under stands language through tok en isation .\n\nAs a rough rule of thumb: 100 tokens is about 75 words, or a short paragraph. A typical novel runs around 100,000 words — that's roughly 133,000 tokens. Claude's extended context window can hold the equivalent of several books at once.\nThe context window: Claude's working memory\n\nEvery conversation with Claude happens inside a context window — a fixed-size buffer that holds everything Claude can \"see\" at once. This includes your entire conversation history, any documents you paste in, system instructions, and Claude's own responses.\n\nOnce the window fills up, older content scrolls out of view. Claude doesn't forget it in the human sense — it simply can't read past what fits. This is why very long conversations can occasionally feel like Claude loses track of something said much earlier."
url: "https://hashnode.com/posts/what-are-tokens-and-why-should-you-care/6a0d967ab1ed7bce01c42344"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [What Are Tokens — And Why Should You Care?](https://vishalmathur.in/what-are-tokens-and-why-should-you-care?utm_source=hashnode&utm_medium=feed)

The hidden unit of measurement that shapes every conversation you have with Claude. You type a message to Claude. You hit send. A response flows back in seconds. Simple, right? But beneath that seamless exchange, something interesting is happening — your words are being sliced into tiny linguistic units called tokens before Claude ever "reads" them. Tokens are the atomic unit of language for large language models. They're not characters, and they're not always full words. They sit somewhere in between — and understanding them unlocks a clearer picture of how AI language models actually work, why they have limits, and how to work with those limits instead of against them. So what exactly is a token? Think of tokenisation as breaking text into the most useful chunks for a model to learn from. Common words like "the" or "and" are usually one token. Longer or rarer words might get split into two or three pieces. Punctuation, spaces, and newlines all count too. Example — how this sentence gets tokenised Claude under stands language through tok en isation . As a rough rule of thumb: 100 tokens is about 75 words, or a short paragraph. A typical novel runs around 100,000 words — that's roughly 133,000 tokens. Claude's extended context window can hold the equivalent of several books at once. The context window: Claude's working memory Every conversation with Claude happens inside a context window — a fixed-size buffer that holds everything Claude can "see" at once. This includes your entire conversation history, any documents you paste in, system instructions, and Claude's own responses. Once the window fills up, older content scrolls out of view. Claude doesn't forget it in the human sense — it simply can't read past what fits. This is why very long conversations can occasionally feel like Claude loses track of something said much earlier.

May 20 · 3 min read · vishalmathur.in  
Read the article

-   [#tokens](https://hashnode.com/tag/tokens)
-   [#claude](https://hashnode.com/tag/claude)
-   [#ai](https://hashnode.com/tag/ai)
-   [#ai-tools](https://hashnode.com/tag/ai-tools)

## Written by

[Vishal Mathur](https://hashnode.com/@vishalmathur)

@vishalmathur · India

  
[View profile](https://hashnode.com/@vishalmathur)

Hi! I am Vishal Mathur, an Ethical Hacker, Developer, IT Consultant, SYS Admin and more. Check out the resources I offer below and sign up f

With over 9 years of experience as in IT, I have led technology operations across diverse industries, ensuring robust IT infrastructure, network security, and team development. My expertise spans managing IT infrastructure &amp; operations, IT policy, and backup/disaster recovery. My expertise also includes IT asset management, Google Workspace &amp; Office 365, endpoint security, DLP, and cross-platform systems (Windows/Linux/Mac OS) etc. Additionally, I hold certifications in Google IT Support, CCNA, and IBM Cybersecurity, reinforcing my commitment to continuous learning and delivering robust technology solutions. Thank you for your time and consideration. Best regards, Vishal Mathur

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
