How do you refrain from refactoring?

Visualised as a chart, my daily productivity would start with a nice steady incline, but would plummet as soon as I see a piece of code which could be written significantly better or that we're using a dependency that's a major version behind. After I'm done refactoring, I often feel like the changes weren't necessary (yet), that I was just satisfying my obsession.

In other words, I feel like my productivity gets worse the more I know, which is a reality that I'm refusing to accept. 😄 I believe that ignoring code unrelated to our task is a skill that can be learned.

How do you refrain yourself from refactoring?