Puzzle: By adding 1 word to the vocabulary and changing the vectors and weights, build an example that illustrates the need for position embedding.
Open the Colab notebook linked at the top of this post to try it.
Comments Welcome
+Hint for the Puzzle: "ignores" is the word. Constructing this helps you see why you need residuals, position embedding, and more layers!
Open the Colab notebook linked at the top of this post to try it.
Give-away: The word to add to the vocabulary is an optional disobey or ignore - modifies verb+attribute tokens. This takes the total vocabulary to 9 words. Add 7-bits of position one-hot. Add 1 extra layer 2 heads each. A good exercise to strengthen the intuition.
Tech-Aarvam
Puzzle: By adding 1 word to the vocabulary and changing the vectors and weights, build an example that illustrates the need for position embedding.
Open the Colab notebook linked at the top of this post to try it.
Comments Welcome