The one-trillion versus eight-trillion token comparison is most useful when kept attached to the in-context-learning tasks. Your knowledge-retention section reports parity, so the article should not leave readers expecting an eightfold advantage across every capability.
I would also separate training-data efficiency from compute efficiency. Different architectures can spend different amounts of computation per token, and associative memory can move costs into lookup or storage. A compact table with task scores, training compute, and inference memory would make the smaller-data result easier to interpret. Learning from examples in the prompt is another distinct claim from permanently updating model weights.
Ahmet Özel
The one-trillion versus eight-trillion token comparison is most useful when kept attached to the in-context-learning tasks. Your knowledge-retention section reports parity, so the article should not leave readers expecting an eightfold advantage across every capability.
I would also separate training-data efficiency from compute efficiency. Different architectures can spend different amounts of computation per token, and associative memory can move costs into lookup or storage. A compact table with task scores, training compute, and inference memory would make the smaller-data result easier to interpret. Learning from examples in the prompt is another distinct claim from permanently updating model weights.