---
title: "Discussion on \"Embedding is just a vector\" | Hashnode"
description: "Discussion on \"Embedding is just a vector\". Token embeddings are rows of a matrix, looked up by an integer. What is the vector, why similarity uses angles, & why embedding layer is context-free"
url: "https://hashnode.com/posts/embedding-is-just-a-vector/6ac31c2ce5fdec84e1b8951b"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Embedding is just a vector](https://monthlydatascience.hashnode.dev/embedding-is-just-a-vector?utm_source=hashnode&utm_medium=feed)

Token embeddings are rows of a matrix, looked up by an integer. What is the vector, why similarity uses angles, & why embedding layer is context-free

Oct 5 · 6 min read · monthlydatascience.hashnode.dev  
Read the article

-   [#embeddings](https://hashnode.com/tag/embeddings)
-   [#ai](https://hashnode.com/tag/ai)
-   [#llm](https://hashnode.com/tag/llm)

## Written by

[Mohitm](https://hashnode.com/@mohitm)

@mohitm · Bangalore, India

  
[View profile](https://hashnode.com/@mohitm)

Data science professional with around 15 years experience, working to on Pricing and Media analytics practice for CPG companies. Currently building a micro-learning app for Stats/ ML learning. Exploring the AI goodness (and the slop) like the rest.

## Responses 3

Markdown supported · ⌘↵ to post

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 5](https://hashnode.com/posts/embedding-is-just-a-vector/6ac31c2ce5fdec84e1b8951b/comment/6ac39ab4147f64827744320c)

The row-lookup explanation makes the static token embedding easy to understand. I would qualify the claim that magnitude carries no meaning, though: whether vector norm is useful depends on the training objective and the similarity function the particular model was trained to use.

For a retrieval model, normalizing a dot-product-trained representation can change rankings rather than merely speed up the same calculation. A small example with two vectors sharing a direction but having different lengths would show that difference directly. Keeping token-table embeddings, contextual hidden states and pooled sentence embeddings separate would also help readers avoid applying the lookup-table explanation unchanged to the vectors they store in a retrieval index.

[Mohitm](https://hashnode.com/@mohitm)

[Oct 6](https://hashnode.com/posts/embedding-is-just-a-vector/6ac31c2ce5fdec84e1b8951b/comment/6ac39ab4147f64827744320c#reply-6ac47c4c38f2a597f7e9dd95)

you're right that "magnitude carries no meaning" is too broad. It holds for the static lookup table under cosine similarity, but whether norm matters really depends on the training objective and similarity function, and the retrieval case proves it: L2-normalizing a dot-product-trained representation can reorder results, not just speed up the same ranking.

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 6](https://hashnode.com/posts/embedding-is-just-a-vector/6ac31c2ce5fdec84e1b8951b/comment/6ac39ab4147f64827744320c#reply-6ac5391c4b05a38b6d5f8a07)

Yes, the similarity function is the key qualification. Even for a static lookup table, cosine ignores norm by definition; that does not establish that the norm contains no information for another use of the same representation.

A compact teaching example could show cosine and raw dot-product scores side by side for one query and two candidates. Then readers can see exactly which rankings normalization preserves and which it changes, before moving from token-table rows to contextual or pooled retrieval embeddings.