---
title: "Discussion on \"Batch Size and LLM Inference Efficiency\" | Hashnode"
description: "Discussion on \"Batch Size and LLM Inference Efficiency\". Having an optimal Batch size can decrease your models cost per token at the time of inference.\n\nThis will be an explanation on how Batch size affects"
url: "https://hashnode.com/posts/batch-size-and-llm-inference-efficiency/6a60ed36b9324f0f504e8b9c"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Batch Size and LLM Inference Efficiency](https://swaritshukla.hashnode.dev/batch-size-and-llm-inference-efficiency?utm_source=hashnode&utm_medium=feed)

Having an optimal Batch size can decrease your models cost per token at the time of inference. This will be an explanation on how Batch size affects

Jul 22 · 4 min read · swaritshukla.hashnode.dev  
Read the article

-   [#AI](https://hashnode.com/tag/AI)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#machinelearning](https://hashnode.com/tag/machinelearning)
-   [#llm](https://hashnode.com/tag/llm)
-   [#llms](https://hashnode.com/tag/llms)
-   [#mlops](https://hashnode.com/tag/mlops)
-   [#ai-inference](https://hashnode.com/tag/ai-inference)
-   [#ai-inference-optimization](https://hashnode.com/tag/ai-inference-optimization)

## Written by

[Swarit Shukla](https://hashnode.com/@swaritshukla)

@swaritshukla

  
[View profile](https://hashnode.com/@swaritshukla)

making machines think

Hey, I am an AI developer. I like to talk about AI and NLP stuff. swaritshukla.me

## Responses 1

Markdown supported · ⌘↵ to post

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Jul 23](https://hashnode.com/posts/batch-size-and-llm-inference-efficiency/6a60ed36b9324f0f504e8b9c/comment/6a622357803339683c1805cb)

Modeling latency as max of compute time and memory time is the framing that finally makes the memory-bound versus compute-bound crossover intuitive. At small batch sizes you're paying for weight loads that serve almost nobody, which is why throughput jumps so much as you fill the batch. Where did the crossover land for the model you tested, and how much did KV cache growth pull it back down?
