Honestly, this is the right instinct. I did almost exactly this with a Go service last year. Spent time "cleaning up" the codebase while an actual performance regression sat in production for weeks.
The thing is, refactoring feels productive because it is productive in some sense. You're shipping code. You're improving things. But improving what, exactly? If users don't notice and bugs pile up, you're just optimizing for developer comfort on someone else's dime.
My rule now: only refactor if it either unlocks a feature you actually need to build or it's actively preventing you from fixing bugs. Otherwise it's procrastination with better optics.
The gnarly form component? Leave it alone if it works. When you need to add a field or change behavior, then clean it up as part of that work. You'll refactor with purpose instead of theater.