As long as you are using some form of module bundler (webpack/browserify/rollup, etc), you can go a really long way with relatively plain JS... The only reason I mention using a bundler is it allows you to write your code in discrete modules.
As it is, you can go a very long way. Also, it makes it easier to add micro libraries via npm after you've done this, which allows you to stay very close to vanilla while still being able to support a growing codebase.