
The question: a 1B-parameter model, trained with AdamW in bf16 mixed precision on 8×A100-40GB. Does it fit under plain DDP? ZeRO-1? ZeRO-2? ZeRO-3? And what micro-batch size can each one handle? Assum
Markdown supported · ⌘↵ to post
No responses yet.