Understanding LLM Embeddings from Scratch — Part 4: How Does an Embedding Actually Learn?
So far, we've learned why embeddings exist, how vectors, dimensions, and similarity work, how nn.Embedding stores vectors and retrieves them using token IDs, how padding and attention masks handle var
Responses
Markdown supported · ⌘↵ to post
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The before/after cosine values make the learning step tangible. I would also compare the predicted next-token distributions for cat and dog across a few random seeds, because the objective constrains those predictions more directly than it constrains a unique embedding geometry.
With a trainable output matrix, two runs can solve the tiny task while arranging their internal coordinates differently. Showing low prediction disagreement alongside the cosine measurements would make the result more robust pedagogically: the shared predictive role is learned, while the exact vector arrangement remains a property of the fitted model.