---
title: "#tiktokenizer articles | Hashnode"
description: "Explore 5 articles tagged #tiktokenizer on Hashnode. Tutorials, guides, and how-tos from the community."
url: "https://hashnode.com/tag/tiktokenizer"
---

Tag

# #tiktokenizer

5 posts · 0 followers

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 11, 2025](https://hashnode.com/posts/use-the-tiktoken-package-to-tokenize-text-for-openai-llms/68712dffec2ae6f509708c16)

### [Use the tiktoken package to tokenize text for OpenAI LLMs.](https://thinkthroo.com/blog/use-the-tiktoken-package-to-tokenize-text-for-openai-llms?utm_source=hashnode&utm_medium=feed)

In this article, you will learn how to use the tiktoken package to tokenize text for OpenAI LLMs. We will look at: What is tiktoken? tiktoken usage examples in Tiktokenizer app. What is tiktoken? If you need a programmatic interface for tokenizi...

2 min read · [thinkthroo.com](https://thinkthroo.com/blog/use-the-tiktoken-package-to-tokenize-text-for-openai-llms?utm_source=hashnode&utm_medium=feed)

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 10, 2025](https://hashnode.com/posts/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-4/686fdc90331293d578fe1921)

### [Here's how OpenAI Token count is computed in Tiktokenizer - Part 4](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-4?utm_source=hashnode&utm_medium=feed)

In this article, we will review how OpenAI Token count is computed in Tiktokenizer. We will look at: Text preview Token IDs preview Text preview When you write some message in https://tiktokenizer.vercel.app/, this application gives you the toke...

2 min read · [thinkthroo.com](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-4?utm_source=hashnode&utm_medium=feed)

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 9, 2025](https://hashnode.com/posts/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-3/686e8b06fbb49b0ad25c9379)

### [Here's how OpenAI Token count is computed in Tiktokenizer - Part 3](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-3?utm_source=hashnode&utm_medium=feed)

In this article, we will review how OpenAI Token count is computed in Tiktokenizer — Part 3. We will look at: OpenSourceTokenizer class For more context, read part 2. OpenSourceTokenizer class In tiktokenizer/src/models/tokenizer.ts, at line 82,...

2 min read · [thinkthroo.com](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-3?utm_source=hashnode&utm_medium=feed)

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 8, 2025](https://hashnode.com/posts/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-2/686d39a6fc15e5bc0299b4bc)

### [Here's how OpenAI Token count is computed in Tiktokenizer - Part 2](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-2?utm_source=hashnode&utm_medium=feed)

In this arcticle, we will review how OpenAI Token count is computed in Tiktokenizer. We will look at: Tokenizer interface TiktokenTokenizer For more context, read the part 1. Tokenizer interface In tiktokenizer/src/models/tokenizer.ts, at lin...

3 min read · [thinkthroo.com](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-2?utm_source=hashnode&utm_medium=feed)

[Ramu Narasinga](https://hashnode.com/@ramunarasinga) · [Jul 7, 2025](https://hashnode.com/posts/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-1/686be81edfe2948d01c3cd2a)

### [Here's how OpenAI Token count is computed in Tiktokenizer - Part 1](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-1?utm_source=hashnode&utm_medium=feed)

In this article, we will review how OpenAI token count is computed in tiktokenizer. We will look at: TokenViewer component tokenizer createTokenizer function TokenViewer component In tiktokenizer/src/pages/index.ts, at line 86, you will find th...

2 min read · [thinkthroo.com](https://thinkthroo.com/blog/heres-how-openai-token-count-is-computed-in-tiktokenizer-part-1?utm_source=hashnode&utm_medium=feed)
