---
title: "Discussion on \"Advanced Transformers: Positional Encodings (RoPE) & Efficient Attention Mechanisms (KV Caching, MQA, GQA, MLA)\" | Hashnode"
description: "Discussion on \"Advanced Transformers: Positional Encodings (RoPE) & Efficient Attention Mechanisms (KV Caching, MQA, GQA, MLA)\". In modern Large Language Model (LLM) architectures, two foundational engineering bottlenecks dictate model scalability and inference latency:\n\nRepresenting Order: How self-attention mechanisms—which a"
url: "https://hashnode.com/posts/advanced-transformers-positional-encodings-rope-efficient-attention-mechanisms-kv-caching-mqa-gqa-mla/6a9f04585a20161fc07430bd"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Advanced Transformers: Positional Encodings (RoPE) & Efficient Attention Mechanisms (KV Caching, MQA, GQA, MLA)](https://rahul-ai.hashnode.dev/advanced-transformers-positional-encodings-rope-efficient-attention-mechanisms-kv-caching-mqa-gqa-mla?utm_source=hashnode&utm_medium=feed)

In modern Large Language Model (LLM) architectures, two foundational engineering bottlenecks dictate model scalability and inference latency: Representing Order: How self-attention mechanisms—which a

Sep 7 · 15 min read · rahul-ai.hashnode.dev  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#artificial-intelligence](https://hashnode.com/tag/artificial-intelligence)
-   [#artificialintelligence](https://hashnode.com/tag/artificialintelligence)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#machinelearning](https://hashnode.com/tag/machinelearning)
-   [#deeplearning](https://hashnode.com/tag/deeplearning)
-   [#deep-learning](https://hashnode.com/tag/deep-learning)
-   [#transformers](https://hashnode.com/tag/transformers)
-   [#kv-cache](https://hashnode.com/tag/kv-cache)
-   [#kvcache](https://hashnode.com/tag/kvcache)
-   [#attention-mechanism](https://hashnode.com/tag/attention-mechanism)
-   [#attention](https://hashnode.com/tag/attention)
-   [#mqa](https://hashnode.com/tag/mqa)
-   [#positional-encoding](https://hashnode.com/tag/positional-encoding)

## Written by

[Rahul Sai Indeevar V](https://hashnode.com/@rahul-ai)

@rahul-ai · India

  
[View profile](https://hashnode.com/@rahul-ai)

## Responses 3

Markdown supported · ⌘↵ to post

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Sep 11](https://hashnode.com/posts/advanced-transformers-positional-encodings-rope-efficient-attention-mechanisms-kv-caching-mqa-gqa-mla/6a9f04585a20161fc07430bd/comment/6aa3e767fc47a7d4fe4bad7d)

@Rahul Sai Indeevar V - adding this here since the reply box on your thread would not submit for me.

If you do write that update, the experiment that makes the prefill and decode split click is measuring both phases separately at two or three batch sizes. Prefill throughput scales with batch until you saturate compute; decode throughput barely moves, because you re-read the whole KV cache either way. Seeing those two curves next to each other explains why continuous batching exists, and why a single tokens-per-second figure hides most of what is actually happening.

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Sep 7](https://hashnode.com/posts/advanced-transformers-positional-encodings-rope-efficient-attention-mechanisms-kv-caching-mqa-gqa-mla/6a9f04585a20161fc07430bd/comment/6a9f11d3113e6fc232d4a315)

Putting RoPE and the KV cache family in one article makes sense because they are the two things that decide what long context actually costs you. One framing worth adding for readers coming from the serving side: MQA and GQA are memory bandwidth optimisations more than compute ones. Decoding is bandwidth-bound - you re-read the whole cache every token - so shrinking the number of KV heads speeds up generation even when FLOPs barely move, which surprises people who benchmark on prefill and see nothing. The practical consequence of the cache section is that serving economics stop being about tokens per second and start being about how many contexts fit in memory at once. Long-running agent workloads keep large caches resident across many steps, so cost per resident context-hour predicts your bill better than cost per token does. Worth noting on RoPE too that extending context at inference is not free - the positions the model never trained on behave differently, which is why scaling tricks exist and why long-context benchmarks disagree with each other so much.

[Rahul Sai Indeevar V](https://hashnode.com/@rahul-ai)

[Sep 8](https://hashnode.com/posts/advanced-transformers-positional-encodings-rope-efficient-attention-mechanisms-kv-caching-mqa-gqa-mla/6a9f04585a20161fc07430bd/comment/6a9f11d3113e6fc232d4a315#reply-6a9f961e12b2192ca9b136b5)

Thank you for your breakdown! While this article was just a projection of my current knowledge, I’ll definitely highlight this distinction in a future update. Thanks for adding so much value to the post!" Today, I got to learn something new! 'The distinction between the prefill phase (compute-bound, parallelized prompt processing) and the decoding phase (memory-bandwidth bound, memory-bound autoregressive token generation) is where so many benchmarking comparisons fall flat'. Looking forward to learn more.