You don't "stumble" upon refactoring. In modern days, refactoring is a first class citizen in the development industry, exactly like unit testing (and other good practices).
You should have a time slot or "ticket" for it, say 1 hour a day. If some days you don't need to refactor, you just keep putting tickets aside, until you need them.
At this point you spend some or all of this accumulated time and do your refactoring. This helps with "time boxing" and development schedule in general.