---
title: "Discussion on \"Does a 1B Model Fit on 8×A100-40GB? \nMemory Arithmetic for DDP and ZeRO 1/2/3\" | Hashnode"
description: "Discussion on \"Does a 1B Model Fit on 8×A100-40GB? \nMemory Arithmetic for DDP and ZeRO 1/2/3\". The question: a 1B-parameter model, trained with AdamW in bf16 mixed precision on 8×A100-40GB. Does it fit under plain DDP? ZeRO-1? ZeRO-2? ZeRO-3? And what micro-batch size can each one handle?\nAssum"
url: "https://hashnode.com/posts/does-a-1b-model-fit-on-8-a100-40gb-memory-arithmetic-for-ddp-and-zero-1-2-3/6afd1027818f8fb17f206ada"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Does a 1B Model Fit on 8×A100-40GB? Memory Arithmetic for DDP and ZeRO 1/2/3](https://dpanshi.hashnode.dev/does-a-1b-model-fit-on-8-a100-40gb-memory-arithmetic-for-ddp-and-zero-1-2-3?utm_source=hashnode&utm_medium=feed)

The question: a 1B-parameter model, trained with AdamW in bf16 mixed precision on 8×A100-40GB. Does it fit under plain DDP? ZeRO-1? ZeRO-2? ZeRO-3? And what micro-batch size can each one handle? Assum

1d ago · 5 min read · dpanshi.hashnode.dev  
Read the article

## Written by

[Deepanshi](https://hashnode.com/@dpanshi)

@dpanshi · San F

  
[View profile](https://hashnode.com/@dpanshi)

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
