Jul 11, 2025 · 2 min read · In this article, you will learn how to use the tiktoken package to tokenize text for OpenAI LLMs. We will look at: What is tiktoken? tiktoken usage examples in Tiktokenizer app. What is tiktoken? If you need a programmatic interface for tokenizi...
Join discussionJul 10, 2025 · 2 min read · In this article, we will review how OpenAI Token count is computed in Tiktokenizer. We will look at: Text preview Token IDs preview Text preview When you write some message in https://tiktokenizer.vercel.app/, this application gives you the toke...
Join discussionJul 9, 2025 · 2 min read · In this article, we will review how OpenAI Token count is computed in Tiktokenizer — Part 3. We will look at: OpenSourceTokenizer class For more context, read part 2. OpenSourceTokenizer class In tiktokenizer/src/models/tokenizer.ts, at line 82,...
Join discussionJul 8, 2025 · 3 min read · In this arcticle, we will review how OpenAI Token count is computed in Tiktokenizer. We will look at: Tokenizer interface TiktokenTokenizer For more context, read the part 1. Tokenizer interface In tiktokenizer/src/models/tokenizer.ts, at lin...
Join discussionJul 7, 2025 · 2 min read · In this article, we will review how OpenAI token count is computed in tiktokenizer. We will look at: TokenViewer component tokenizer createTokenizer function TokenViewer component In tiktokenizer/src/pages/index.ts, at line 86, you will find th...
Join discussion
