I see three different options:

1. Do not support IE anymore. It's an old browser and Microsoft is pushing people onto Windows 10 with the Edge browser. Edge does have modern ES6 built-in. This is not always an option.
2. Use an available polyfill, like core-js, which is the most complete and battle-tested polyfill you can get. It's modular, so you can only pick what you need. However, you still do not get new language features, like arrow functions, template strings, and so on. It's technically impossible to polyfill them.
3. Use a transpiler. Personally, I am quite against the usage of transpilers, since that means you always have a build-step before deploying code and they are often miss-used for unfinished specs. The transpilation of ES6 to ES5 or ES3, however, is a legitimate use-case imho and you can use it to support older browsers. If I had to implement a transpiled version for older browsers, I would also implement a browser-switch, which only delivers the transpiled code to older and unknown browsers, so you can use modern, optimized features on newer browsers.

Older version of IE not only lack support for modern JavaScript features, but also have much slower JavaScript engines in them. If you're trying to polyfill new JavaScript features in old browsers using JavaScript, I feel like it will slow down how fast all of your code can run - since it can only run as fast as can be interpreted through all those polyfills.

If you have code written as ES6 and you would like to run it in older browsers, could a solution like using Babel to transpile that code down JavaScript written using features that older browsers understand (before loading it in the browser) lead to better performance in those older browsers?

I have all the main polyfils I needed in BunnyJS

You may download only what you need or just polyfills.min.js from CDN