TGive-away: The word to add to the vocabulary is an optional disobey or ignore - modifies verb+attribute tokens. This takes the total vocabulary to 9 words. Add 7-bits of position one-hot. Add 1 extra layer 2 heads each. A good exercise to strengthen the intuition.Reply·Article·6d ago·Attention is simpler than you think
T+Hint for the Puzzle: "ignores" is the word. Constructing this helps you see why you need residuals, position embedding, and more layers! Open the Colab notebook linked at the top of this post to try it. Comments WelcomeReply·Article·Sep 10·Attention is simpler than you think
TPuzzle: By adding 1 word to the vocabulary and changing the vectors and weights, build an example that illustrates the need for position embedding. Open the Colab notebook linked at the top of this post to try it. Comments WelcomeComment·Article·Sep 9·Attention is simpler than you think
