Personally I am always targeting the browser environment whenever I write HTML, CSS, or JavaScript which means in the past I was limited to ES5.
I didn't see much use in writing ES6 only to have to transpile it down to much more ES5 that I wouldn't even be seeing and trust that if there were any bugs or problems I'd be able to figure out if it was my compile that was bad, my tools, or my own source code, etc. I don't see the advantage to targeting ahead of browser support for the browsers you are writing for.
However, as time moves on and browsers gain many ES6 features natively, we should be adopting these features and not transpiling them. The reason? Because newer ES6 features, even when they look like a little bit of syntactic sugar for something in ES5 have usually been designed in a way that's more conducive to optimization.
If you transpile these ES6 features down to ES5 for any browser that supports the ES6 feature it will still work, you're just doing extra work to actively throw away the recent optimizations we've been able to make.
So - you shouldn't be afraid of, or shy about using new ES6 features, but keep in mind the optimizations available for the features that are supported in the browsers you are targeting :D