How to Train Compute-Optimal Large Language Models?

Introduction Recently, an LLM with only 70B parameters outperforms GPT 3. This LLM, called Chinchilla, was developed by Hoffmann and his colleagues. In their work, they state that current LLMs are not compute-optimal. Why is this? How do they train t...