Rotary Positional Embeddings (RoPE)

Today, we are tackling a different problem: How does a LLM know where words are? Transformers, by design, are permutation invariant. Without explicit help, they view a beautifully structured sentence