I think "hacker" is just one of those words that got used differently by the media and the public than by the people who actually knew anything about it.
That caused most people to think of "digital burglar" when they hear "hacker". And that made the "ethical" prefix necessary.
You can try to fight such a use, if you feel it's worth the energy trying to convince millions of people who don't really care enough to listen, but I don't recommend it.
What does your grandma thinks a "pen tester" is?