You have to define what is acceptable and what is not. What increase 'code quality' and what is not.
I have a rule of thumb for that: code.test > code.readability > code.robustness > doc.readability > feature > code.performance
(test increase code quality more than readability. Readability increase code quality more than robustness...)
And read it like that : If I can increase the performance of the code without decrease the feature and the doc and the robustness and the code's readability and the test cover, so I should do it.
Again, if I can increase the robustness but the code's readability or the test will suffer, I should not do it.
This rule should be created and agreed by the whole team.
Hope it can help.
Cheers 🍻