+Hint for the Puzzle: "ignores" is the word. Constructing this helps you see why you need residuals, position embedding, and more layers! Open the Colab notebook linked at the top of this post to try it.

Comments Welcome

Give-away: The word to add to the vocabulary is an optional disobey or ignore - modifies verb+attribute tokens. This takes the total vocabulary to 9 words. Add 7-bits of position one-hot. Add 1 extra layer 2 heads each. A good exercise to strengthen the intuition.