The Silicon Sorting Hat: Demystifying CPU, GPU, and TPU Architectures

Thanks for your reply, Karthik. Here is my response to this:

When you're trying to decide whether to run your LLM inference on TPUs or GPUs, it really comes down to a few practical trade-offs: how predictable your batches are, your model's shape, and the distinct math behind the prefill and decode phases.

If your workload is built around steady, high-volume traffic with fixed batch sizes and uniform sequence lengths, TPUs are absolute powerhouses. Thanks to their structured 2D or 3D mesh interconnects, they can crunch matrices at scale with incredible cost-efficiency for closed ecosystems.

On the flip side, if you're dealing with chaotic chat traffic, ragged batching, and dynamic sequence lengths, GPUs give you the adaptability you need. Their flexible NVLink and NVSwitch setups make it much easier to handle custom attention mechanisms and shifting model shapes without hitting performance bottlenecks.

While both architectures do great during the heavy lifting of the compute-bound prefill phase, the memory-bandwidth crunch during decode shifts the focus entirely to memory capacity per dollar and your serving stack. Ultimately, predictable, massive-scale production traffic makes TPUs a clear winner for your wallet, while rapid iteration and unpredictable request patterns make the flexibility of GPUs well worth it.

Are you currently trying to optimise an inference pipeline for a specific model size, or looking at this from a cost-modelling perspective?