Well if I get the chance, I talk to the person who wrote it, to get the full picture. Because it's way to easy to judge or miss something, without getting the circumstance or understand the whole picture.
Certain things I'll always refactor like stated in the pragmatic programmer: Don't leave broken windows.
So naming or comments are getting replaced and changed on the fly all the time or logical conditions inversed to increase the readability.
I try to separate commits for that though so one is about semantics / style and there is no functionality change.
If there is a design flaw or something in the code architecture. It is something I will create a todo for and in my code there is a rule ... in the final commit there are no todos without a ticket number.
because I don't need to read my good intentions 3 years laters and it's documented for the project management to see so there is transparency outside of the codebase as well.
Also for save refactoring you need to have several levels of testing, if those tests don't exist or you have certain database lock / timeout / async changes this can escalate from personal taste of aesthetics to a complete breakdown of the production system because we like to think about or problem localized ... maybe the 'ugly code' is the fully optimized version of twelve system crashes .... which leads us to my first paragraph ... talk to the person who did it and see if you were just hasty.
But yeah in the end it really depends on a lot of different factors ....