---
title: "Discussion on \"Deeper Transformers Are Forgetting What They Learned. MoDA Is the Fix.\" | Hashnode"
description: "Discussion on \"Deeper Transformers Are Forgetting What They Learned. MoDA Is the Fix.\". Deep transformers forget early-layer signals. MoDA fixes this with depth KV caching — better performance, only 3.7% extra compute. Here's how it works"
url: "https://hashnode.com/posts/deeper-transformers-are-forgetting-what-they-learned-moda-is-the-fix/69c41f5610e664c5dacf4a82"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Deeper Transformers Are Forgetting What They Learned. MoDA Is the Fix.](https://abishkamran.hashnode.dev/deeper-transformers-are-forgetting-what-they-learned-moda-is-the-fix?utm_source=hashnode&utm_medium=feed)

Deep transformers forget early-layer signals. MoDA fixes this with depth KV caching — better performance, only 3.7% extra compute. Here's how it works

Mar 25 · 18 min read · abishkamran.hashnode.dev  
Read the article

-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#llm](https://hashnode.com/tag/llm)
-   [#transformers](https://hashnode.com/tag/transformers)
-   [#deep-learning](https://hashnode.com/tag/deep-learning)
-   [#ai](https://hashnode.com/tag/ai)
-   [#AI](https://hashnode.com/tag/AI)
-   [#architecture](https://hashnode.com/tag/architecture)
-   [#nlp](https://hashnode.com/tag/nlp)

## Written by

[Abish Kamran](https://hashnode.com/@AbishKamran)

@AbishKamran

  
[View profile](https://hashnode.com/@AbishKamran)

Web Developer | AI/ML Enthusiast

## Responses

Markdown supported · ⌘↵ to post

No responses yet.