Red Pajama LLM: Public Dataset Analysis Revealed
Key Highlights
Red Pajama 2 is open-source language model pre-training dataset containing a massive 30 trillion tokens, it the largest public dataset for language model pre-training.
The dataset includes over 100 billion text documents from 84 Comm...
