---
title: "Discussion on \"AI Fundamentals for Developers: Tokens, Vectors and Embeddings\" | Hashnode"
description: "Discussion on \"AI Fundamentals for Developers: Tokens, Vectors and Embeddings\". If you're reading about AI, LLMs, or RAG, you'll hear terms like tokens, vectors, and embeddings all the time. They may sound complicated, but the basic idea is quite simple.\nComputers don't understan"
url: "https://hashnode.com/posts/ai-fundamentals-for-developers-tokens-vectors-and-embeddings/6a5dbd0baea38f88c1a47396"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [AI Fundamentals for Developers: Tokens, Vectors and Embeddings](https://rahulvijayvergiya.hashnode.dev/ai-fundamentals-for-developers-tokens-vectors-and-embeddings?utm_source=hashnode&utm_medium=feed)

If you're reading about AI, LLMs, or RAG, you'll hear terms like tokens, vectors, and embeddings all the time. They may sound complicated, but the basic idea is quite simple. Computers don't understan

Jul 20 · 6 min read · rahulvijayvergiya.hashnode.dev  
Read the article

-   [#artificial-intelligence](https://hashnode.com/tag/artificial-intelligence)
-   [#generative-ai](https://hashnode.com/tag/generative-ai)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)

## Written by

[Rahul Vijayvergiya](https://hashnode.com/@rahulvijayvergiya)

@rahulvijayvergiya · Pune, India

  
[View profile](https://hashnode.com/@rahulvijayvergiya)

Engineering Lead | Technical Architect | Full-Stack | AWS | MERN | Node.js | React.js | Typescript | GraphQl | MongoDB | MySQL| PostgreSQL |

I am a dedicated and goal-driven Lead full-stack developer/architecture with over 15 years of experience in leading and designing web application. My expertise spans both front-end, where I specialize in technologies like React/Redux, jQuery, Bootstrap, CSS, and HTML5, and back-end, where I work with AWS, Node.js, Express, and MongoDB. In my approach to web architecture, I prioritize simplicity, efficiency, and resilience, with a keen interest in developing interactive products that seamlessly handle errors and ensure uninterrupted performance. I constantly strive to improve my skills and stay updated with the latest advancements in the field.

## Responses 1

Markdown supported · ⌘↵ to post

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Jul 21](https://hashnode.com/posts/ai-fundamentals-for-developers-tokens-vectors-and-embeddings/6a5dbd0baea38f88c1a47396/comment/6a5f80fe33ae9f47ec3dee10)

The detail that tokens carry leading spaces (" like" vs "like") is one people almost never notice until their token counts stop matching their word counts. Worth adding that this is also why the same string can cost different amounts across models, since each tokenizer has its own vocab and merge rules. Do you plan to cover how embedding dimensionality trades off retrieval quality against index size in a follow-up?
