The one-hot example where cat and dog sit at cosine zero, identical to cat and Tuesday, is the clearest way I've seen to motivate why embeddings had to exist. It makes the progression from bag-of-words to n-grams feel inevitable rather than arbitrary. If you extend the series, showing where static word2vec still fails, like one vector for a word with two meanings, would set up contextual embeddings really well.
That single-vector bottleneck is the exact ceiling static embeddings hit. Moving from treating a word as a fixed vector to making it a function of its context makes the jump to models like BERT feel like an unavoidable next step, and I'll definitely dive deeper into that in future blogs!
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
The one-hot example where cat and dog sit at cosine zero, identical to cat and Tuesday, is the clearest way I've seen to motivate why embeddings had to exist. It makes the progression from bag-of-words to n-grams feel inevitable rather than arbitrary. If you extend the series, showing where static word2vec still fails, like one vector for a word with two meanings, would set up contextual embeddings really well.