Consider the scope of your current work. If you can't pin your changes to that scope, save it for later.
If the refactor is significant, judge its worth. It depends upon how problematic the existing code is. If your refactor makes things easier to change / work with / understand in the future, it's probably worth it.