Understanding LLM Embeddings from Scratch — Part 3: The Embedding Matrix
In Module 1, we learned why embeddings exist. In Module 2, we understood what vectors are, including their dimensions, magnitude, dot products, and cosine similarity. Now, let's connect everything to
Responses
Markdown supported · ⌘↵ to post
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The vocabulary-size memory example assumes the tokenizer and embedding table agree on the token-to-row mapping. That compatibility matters beyond matching the number of rows: a tokenizer with reordered IDs can fit the same matrix shape while retrieving the wrong learned vectors.
I would package the tokenizer revision with the checkpoint and add a tiny known-token-to-ID check when loading custom assets. Extending a vocabulary also needs an explicit embedding resize and initialization step. This turns the table lookup explanation into a useful deployment invariant without requiring readers to understand the later Transformer layers first.