I dislike "specific patterns" as more often than not they shoe-horn you into a specific set of practices or methodologies that may simply not be suited to every task.
There are only three things I ask myself when refactoring:
1) Is this simpler?
2) Is this less code?
3) Will this run faster?
EVERYTHING else is negotiable... and I'm willing to trade #3 in a lot of cases if the task is exceedingly complex. You have to balance ease of maintenance and ability to understand the code at a glance vs. execution time. Squeezing that extra drop of blood from the stone does you no good if in three months you're staring at the code unable to remember how in blazes it even works.
#1 and #2 play to an all important rule I had drilled into me in the '80's.
The less code you use, the less there is to break
A lesson LOST on todays developers it often seems with their wasting 60-100k of HTML to do 8 to 16k's job, half a megabyte of CSS spanning a dozen files to do 24-32k in one file's job, and megabytes of scripting spanning dozens of files on pages that often don't even warrant the PRESENCE of JavaScript!
Or on the server-side, shoe-horning MVC into PHP when it's just not how PHP was actually meant to work. Whilst I appreciate separation of concerns, the MVC model -- whilst well suited to implementing event driven standalone software -- is simply not where I would draw those lines of separation in a pull environment. Hence why so many systems built following MVC end up five to ten times the code needed for even the simplest of tasks.
You see this same level of stupidity when it comes to creating a uniform "standardized" approach to every problem. You end up wasting code shoe-horning things into fitting that model, instead of using the best solution to each problem. You end up like the carpenter who's only tool is a hammer. Suddenly everything looks like a nail.
Just look at the places where posixisms are holding back the industry; not every type of hardware cleanly maps to a serial character device or a filesystem item... so for the love of Christmas STOP TRYING TO FORCE THEM TO! Look at the idiotic client-server model that X-Windows relies upon that really has no place apart from introducing unnecessary overhead when the 'client' and 'server' are running on the same bloody machine!
Don't get hung up on following an exact pattern for everything, all you end up with is a bloated wreck.
That's why for me the bottom line for refactoring: Is it simpler? Is it smaller? Is it faster? No? Then what the **** are you doing?!?