Discussion on "Embedding is just a vector"

Mohitm · 2026-10-05T03:40:28.331Z

Token embeddings are rows of a matrix, looked up by an integer. What is the vector, why similarity uses angles, & why embedding layer is context-free

AÖ

Ahmet Özel

Oct 5

The row-lookup explanation makes the static token embedding easy to understand. I would qualify the claim that magnitude carries no meaning, though: whether vector norm is useful depends on the training objective and the similarity function the particular model was trained to use.

For a retrieval model, normalizing a dot-product-trained representation can change rankings rather than merely speed up the same calculation. A small example with two vectors sharing a direction but having different lengths would show that difference directly. Keeping token-table embeddings, contextual hidden states and pooled sentence embeddings separate would also help readers avoid applying the lookup-table explanation unchanged to the vectors they store in a retrieval index.

Mohitm

Oct 6

you're right that "magnitude carries no meaning" is too broad. It holds for the static lookup table under cosine similarity, but whether norm matters really depends on the training objective and similarity function, and the retrieval case proves it: L2-normalizing a dot-product-trained representation can reorder results, not just speed up the same ranking.

Yes, the similarity function is the key qualification. Even for a static lookup table, cosine ignores norm by definition; that does not establish that the norm contains no information for another use of the same representation.

A compact teaching example could show cosine and raw dot-product scores side by side for one query and two candidates. Then readers can see exactly which rankings normalization preserves and which it changes, before moving from token-table rows to contextual or pooled retrieval embeddings.

Embedding is just a vector

Written by

Responses3

Search Hashnode

Embedding is just a vector

Written by

Responses3