The detail that tokens carry leading spaces (" like" vs "like") is one people almost never notice until their token counts stop matching their word counts. Worth adding that this is also why the same string can cost different amounts across models, since each tokenizer has its own vocab and merge rules. Do you plan to cover how embedding dimensionality trades off retrieval quality against index size in a follow-up?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
The detail that tokens carry leading spaces (" like" vs "like") is one people almost never notice until their token counts stop matching their word counts. Worth adding that this is also why the same string can cost different amounts across models, since each tokenizer has its own vocab and merge rules. Do you plan to cover how embedding dimensionality trades off retrieval quality against index size in a follow-up?