Rishav Paulrpaul.hashnode.dev·Feb 2, 2025Tokenization in GPT: A Deep Dive for EngineersRole of Tokenization in LLMs Tokenization is the gateway through which raw text transforms into a format usable by large language models (LLMs) like GPT. It acts as the bridge between human-readable content and numerical data that models process. Bef...Building GPT from Scratchgpt
Sergey Shishkinserj-aleks.hashnode.dev·Jan 28, 2025arg, args и argvАргумент? Элемент? Параметр? Токен? … Функции, которые могут использоваться только внутри функций с переменным числом аргументов. (с @) arg - возвращает cnt-й оставшийся аргумент. args - возвращает T, когда есть еще аргументы для извлечения из внутре...picolisp
Luna Milleraiandblockchain.hashnode.dev·Jan 28, 2025What Are the Benefits of Tokenizing Real Estate Investment Trusts?Real Estate Investment Trusts (REITs) have long been a popular vehicle for investors looking to gain exposure to real estate markets without having to directly own properties. By pooling capital from multiple investors, REITs offer a way to invest in...Real Estate
Arman Chaudhary6monthstoaimastery.hashnode.dev·Jan 26, 2025Understanding Tokenization: How to Divide Text EffectivelyWhat is Tokenization? A Real-World Analogy Imagine you're trying to teach a foreign language to someone who has never heard it before. How would you break down communication? You'd start by separating words, understanding their individual meanings, a...Tokenization
Barmanjibackend-finnese.hashnode.dev·Jan 21, 2025Little talks on: Mongoose with Pagination and Aggregation Pipelines.This article is about my new app Youtube-Twitter Tokens- Lets talk about tokenization- Why its important? and What is a Token?Tokens are pieces of data that carry just enough information to facilitate the process of determining a user's identity or a...backend
Awais Ahmadblog.itsahmadawais.com·Jan 4, 2025Tokenization with React.js: Creating Scalable Design Systems for Modern ApplicationsCreating scalable, maintainable, and efficient applications is a core goal in today's web development landscape. With the rise of design systems, developers need a robust method to manage design elements that work seamlessly across various platforms....React
KiwiChipcodecompass.hashnode.dev·Dec 25, 2024LLMs 파인튜닝 프로젝트: 제품 설명으로 가격 예측하기 (4)이번 포스트에서는 LLM의 토큰화에 대해서 잠깐 짚고 넘어가겠다. 🔍 코드 설명 def report(item): prompt = item.prompt # 아이템의 프롬프트(설명) 가져오기 tokens = Item.tokenizer.encode(item.prompt) # 프롬프트를 토큰화해 토큰 ID 리스트로 변환 print(prompt) # 원본 프롬프트 출력 print(tokens[-10:]) # 마지막 ...Learning LLMstoken
Mahmoud Hamed Namnamnlp-series.hashnode.dev·Dec 22, 2024Arabic Language Tokenization Explained: Key Concepts and MethodsIntroduction Tokenization is one of the first steps in Natural Language Processing (NLP), where text is divided into smaller units known as tokens. These units can be words, sentences, or even characters. Tokenization is essential for text analysis, ...4 likes·118 readsfarasa
Bittu Sharmabittublog.hashnode.dev·Dec 19, 2024Exploring Tokenization: Key Facts and InsightsTokenization is a fundamental process in Natural Language Processing (NLP) that involves breaking down a stream of text into smaller units called tokens. These tokens can range from individual characters to full words or phrases, depending on the lev...Tokenization
Vishalvishalchhadekar.hashnode.dev·Dec 18, 2024Optimizing Prompt Writing for Cost-Efficiency in OpenAI ModelsIntroduction OpenAI's advanced language models, such as GPT-4 and GPT-3.5, are powerful tools that process text prompts and generate insightful responses. However, these models operate on a token-based pricing system, where each interaction incurs a ...32 readsAI