Bits Don't Lie: Datatypes in Modern LLMs

May 30, 2025 · 6 min read · Let’s talk about the different datatypes that are being used in modern LLMs like GPT, LLaMa and the like. The most common ones that you might have heard: FP32, BFloat16, Float16, INT8, etc. These are all standard data types and available in PyTorch a...