Oct 20, 2025 · 10 min read · Tokenisation From Scratch: Building the Bridge Between Text and AI Understanding the translation layer that powers every large language model When you type a message to ChatGPT, Llama, or any other language model, something magical happens behind the...
Join discussionJul 22, 2025 · 2 min read · Tiktoken is one of the most popular tokenizers out there. This is a really nice and simple cookbook that shows how to use it. Recently I was optimizing our token counting function that is often used to chunk data to send to embedding models(need prec...
Join discussion
Jul 11, 2025 · 2 min read · In this article, you will learn how to use the tiktoken package to tokenize text for OpenAI LLMs. We will look at: What is tiktoken? tiktoken usage examples in Tiktokenizer app. What is tiktoken? If you need a programmatic interface for tokenizi...
Join discussionJul 10, 2025 · 2 min read · In this article, we will review how OpenAI Token count is computed in Tiktokenizer. We will look at: Text preview Token IDs preview Text preview When you write some message in https://tiktokenizer.vercel.app/, this application gives you the toke...
Join discussionJul 9, 2025 · 2 min read · In this article, we will review how OpenAI Token count is computed in Tiktokenizer — Part 3. We will look at: OpenSourceTokenizer class For more context, read part 2. OpenSourceTokenizer class In tiktokenizer/src/models/tokenizer.ts, at line 82,...
Join discussionJul 8, 2025 · 3 min read · In this arcticle, we will review how OpenAI Token count is computed in Tiktokenizer. We will look at: Tokenizer interface TiktokenTokenizer For more context, read the part 1. Tokenizer interface In tiktokenizer/src/models/tokenizer.ts, at lin...
Join discussionNov 12, 2024 · 8 min read · In this project, we're building a Similarity Search Application that leverages text embeddings to identify and display items similar to a user-provided word. Specifically, this app provides suggestions for words or objects that are related to an inpu...
Join discussionOct 1, 2023 · 6 min read · Introduction OpenAI does have the usage field in its non-stream response. For example with a non-stream request like this (stream: false): { "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": ...
Join discussion