Older version of IE not only lack support for modern JavaScript features, but also have much slower JavaScript engines in them. If you're trying to polyfill new JavaScript features in old browsers using JavaScript, I feel like it will slow down how fast all of your code can run - since it can only run as fast as can be interpreted through all those polyfills.
If you have code written as ES6 and you would like to run it in older browsers, could a solution like using Babel to transpile that code down JavaScript written using features that older browsers understand (before loading it in the browser) lead to better performance in those older browsers?