H2O KV Cache - An Easy Explanation
Heavy Hitter Oracle, the basic idea behind it is that, it retains the tokens which have better attention and evicts the tokens which don't, it also keeps the recent tokens in cache and that is how it
tech.peculiarvivek.com5 min read