Is it the model or the data that's low rank?

I mentioned this little bit of analysis that I recently did during the Latent Space Paper Club, and got a lot of positive feedback, so I did a quick writeup. The recently released Apple Intelligence Foundation Language Models paper has spark a lot of...