From the user side, does the app work? And are those users generally satisfied?
If the answer to both of those questions is "yes," then a wholesale rewrite could bring about issues that could put those users in jeopardy in terms of satisfaction and loyalty.
Couple that with the fact that rewrites ALWAYS take longer than estimated, the safest bet is to refactor in chunks, either as you jump in to that code to add a new feature, or as a stand alone "let's refactor" issue.