Do Transformers Understand Order ?

Self-attention is incredibly powerful. It can compare every word with every other word in parallel but it has one surprising blind spot, it has no inherent understanding of sequence order. Why Is Atte