In the past when I wanted to have smooth scrolling on a website I'd just use this jQuery snippet by CSS-Tricks. It does exactly what I want and didn't require any javascript library (except jQuery).
These days I prefer to avoid using big libraries like jQuery too much, because I only use 5% of the functionalities while getting worse load times. Causing me to look for a vanilla JS version of the jQuery snippet provided by CSS-Tricks.
Is there an elegant way of implementing smooth scrolling by using vanilla JS?
I don't have a specific answer to your question, but I do wonder whether the fact you are seeing increased load times is a product of insufficient caching or something, since jQuery isn't a particularly big library (~32k or thereabouts minified and gzipped).
Jonathan
Breaking things by touching
Have you taken a look at smooth-scroll. It's a plain JS implementation. You may also want to take a look at the code on GitHub.