---
title: "#tiktoken articles | Hashnode"
description: "Explore 8 articles tagged #tiktoken on Hashnode. Tutorials, guides, and how-tos from the community."
url: "https://hashnode.com/tag/tiktoken"
---

Tag

# #tiktoken

8 posts · 0 followers

[Pornima Gaikwad](https://hashnode.com/@pornima1) · [Oct 20, 2025](https://hashnode.com/posts/tokenisation-under-the-hood/68f5f810f355a838b414c424)

### [Tokenisation under the Hood](https://llmtokenizer.hashnode.dev/tokenisation-under-the-hood?utm_source=hashnode&utm_medium=feed)

Tokenisation From Scratch: Building the Bridge Between Text and AI Understanding the translation layer that powers every large language model When you type a message to ChatGPT, Llama, or any other language model, something magical happens behind the...

10 min read · [llmtokenizer.hashnode.dev](https://llmtokenizer.hashnode.dev/tokenisation-under-the-hood?utm_source=hashnode&utm_medium=feed)

[Dhaval Singh](https://hashnode.com/@DhavalSingh) · [Jul 22, 2025](https://hashnode.com/posts/counting-tokens-at-scale-using-tiktoken/687fbe949e43654faf65ec8a)

### [Counting tokens at scale using tiktoken](https://www.dsdev.in/counting-tokens-at-scale-using-tiktoken?utm_source=hashnode&utm_medium=feed)

Using encode\_to\_numpy to count number of tokens in large strings to save memory and CPU

2 min read · [dsdev.in](https://www.dsdev.in/counting-tokens-at-scale-using-tiktoken?utm_source=hashnode&utm_medium=feed)

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 11, 2025](https://hashnode.com/posts/use-the-tiktoken-package-to-tokenize-text-for-openai-llms/68712dffec2ae6f509708c16)

### [Use the tiktoken package to tokenize text for OpenAI LLMs.](https://thinkthroo.com/blog/use-the-tiktoken-package-to-tokenize-text-for-openai-llms?utm_source=hashnode&utm_medium=feed)

In this article, you will learn how to use the tiktoken package to tokenize text for OpenAI LLMs. We will look at: What is tiktoken? tiktoken usage examples in Tiktokenizer app. What is tiktoken? If you need a programmatic interface for tokenizi...

2 min read · [thinkthroo.com](https://thinkthroo.com/blog/use-the-tiktoken-package-to-tokenize-text-for-openai-llms?utm_source=hashnode&utm_medium=feed)

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 10, 2025](https://hashnode.com/posts/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-4/686fdc90331293d578fe1921)

### [Here's how OpenAI Token count is computed in Tiktokenizer - Part 4](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-4?utm_source=hashnode&utm_medium=feed)

In this article, we will review how OpenAI Token count is computed in Tiktokenizer. We will look at: Text preview Token IDs preview Text preview When you write some message in https://tiktokenizer.vercel.app/, this application gives you the toke...

2 min read · [thinkthroo.com](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-4?utm_source=hashnode&utm_medium=feed)

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 9, 2025](https://hashnode.com/posts/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-3/686e8b06fbb49b0ad25c9379)

### [Here's how OpenAI Token count is computed in Tiktokenizer - Part 3](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-3?utm_source=hashnode&utm_medium=feed)

In this article, we will review how OpenAI Token count is computed in Tiktokenizer — Part 3. We will look at: OpenSourceTokenizer class For more context, read part 2. OpenSourceTokenizer class In tiktokenizer/src/models/tokenizer.ts, at line 82,...

2 min read · [thinkthroo.com](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-3?utm_source=hashnode&utm_medium=feed)

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 8, 2025](https://hashnode.com/posts/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-2/686d39a6fc15e5bc0299b4bc)

### [Here's how OpenAI Token count is computed in Tiktokenizer - Part 2](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-2?utm_source=hashnode&utm_medium=feed)

In this arcticle, we will review how OpenAI Token count is computed in Tiktokenizer. We will look at: Tokenizer interface TiktokenTokenizer For more context, read the part 1. Tokenizer interface In tiktokenizer/src/models/tokenizer.ts, at lin...

3 min read · [thinkthroo.com](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-2?utm_source=hashnode&utm_medium=feed)

[Anix Lynch](https://hashnode.com/@anix) · [Nov 12, 2024](https://hashnode.com/posts/langchain-project-3-similarity-search-application/6732faf6d12bcc428be23bbc)

### [Langchain Project 3: Similarity Search Application](https://anixblog.hashnode.dev/langchain-project-3-similarity-search-application?utm_source=hashnode&utm_medium=feed)

Langchain Project 3: Similarity Search Application

8 min read · [anixblog.hashnode.dev](https://anixblog.hashnode.dev/langchain-project-3-similarity-search-application?utm_source=hashnode&utm_medium=feed)

[An Vo](https://hashnode.com/@anvodev) · [Oct 1, 2023](https://hashnode.com/posts/calculate-openai-usage-for-chat-completion-api-stream-in-nodejs/6519009d0d7c53d5b12024cc)

### [Calculate OpenAI usage for Chat Completion API stream in NodeJS](https://anvodev.hashnode.dev/calculate-openai-usage-for-chat-completion-api-stream-in-nodejs?utm_source=hashnode&utm_medium=feed)

Introduction OpenAI does have the usage field in its non-stream response. For example with a non-stream request like this (stream: false): { "model": "gpt-3.5-turbo", "messages": \[ { "role": "user", "content": ...

6 min read · [anvodev.hashnode.dev](https://anvodev.hashnode.dev/calculate-openai-usage-for-chat-completion-api-stream-in-nodejs?utm_source=hashnode&utm_medium=feed)
