True, refactoring is often mentioned as a separate step (after the test passes).
Personally I don't see the point of refactoring your code while you're still implementing.
You could refactor your code after you pass the test and then when you do the next requirement realize it's not going to work, meaning you wasted time.
As I mentioned in the 'reasons', you will write code that can be refactored with confidence, but I would suggest to do this after you're done.
This is personal preference of course.