Yes, I have been scared, but that's during the college days. I used to write PHP, but then I started learning Java and basically loved the language, paradigms etc. I always thought that I would use both the languages so that I didn't forget PHP. However, I didn't get time to go back to PHP and continued writing Java. Eventually, I realised that you don't have to worry about forgetting how to code in a certain language. You should get better with time, learn new and interesting stuff. If you have to code in a certain language, you can always go back and brush-up your skills.
When I was a Java developer I used to believe that this was the language I would use for the next 5-10 years. But things changed, I moved to Node.js and have been working with it for last 3 years.
You never know what language you will end up using in next 5 years. So, master the one you are actively using and learn something new everyday.