---
title: "Discussion on \"Inside Attention: Why Softmax Sits Between QKᵀ and V\" | Hashnode"
description: "Discussion on \"Inside Attention: Why Softmax Sits Between QKᵀ and V\". When I first learned self-attention, the formula looked straightforward:\n$$\\operatorname{Attention}(Q,K,V)=\\operatorname{softmax}\\left(\\frac{QK^\\top}{\\sqrt{d_k}}+M\\right)V$$\nBut one question bothered"
url: "https://hashnode.com/posts/why-do-we-apply-softmax-to-qk-before-multiplying-by-v/6af1b85942416b89d2ff0a96"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Inside Attention: Why Softmax Sits Between QKᵀ and V](https://rounakshow.hashnode.dev/why-do-we-apply-softmax-to-qk-before-multiplying-by-v?utm_source=hashnode&utm_medium=feed)

When I first learned self-attention, the formula looked straightforward: $$\\operatorname{Attention}(Q,K,V)=\\operatorname{softmax}\\left(\\frac{QK^\\top}{\\sqrt{d\_k}}+M\\right)V$$ But one question bothered

2d ago · 6 min read · rounakshow.hashnode.dev  
Read the article

-   [#llm](https://hashnode.com/tag/llm)
-   [#attention-mechanism](https://hashnode.com/tag/attention-mechanism)
-   [#ai](https://hashnode.com/tag/ai)

## Written by

[Rounak Show](https://hashnode.com/@ShowRounak)

@ShowRounak · Remote

  
[View profile](https://hashnode.com/@ShowRounak)

Learning Data Science

Learning Data Science and Sharing the journey through Hashnode.

## Responses

Markdown supported · ⌘↵ to post

No responses yet.