Nah, the impact of your work is more important than your Twitter following! Especially if you're working at a company that will then amplify your work.
Steph on my team had a few hundred followers when I hired her. The quality of her work spoke for itself 😁