Mixtral of Experts: Top-2 Routing Gives 47B Capacity at 13B Active Compute

At roughly 2.08e11 cumulative FLOPs in my own run, a dense baseline lands at 25.31 validation perplexity. At nearly the same compute budget, a sparse MoE lands at 20.98. The absolute delta is 4.324, a