The reason I avoid build tools during development is very much tied to my workflow and ethic. With a transpiler in between me and my code, I can't live debug and change memory on the fly. I've written and recorded videos about this, and it generally boils down to my use of workspaces in Chrome devtools.
I know that you can use sourcemaps to see the original source code, but you can't interact with it as a sourcemap (in particular, you can't change it in real time).
So I've just gotten used to working like this. Also, as fun as ES6 is (the syntax changes), it's really just sugar, and I've been coding for nearly 20 years on the web now, so my fingers are pretty efficient at writing function, so it doesn't really cost me much.
But that is changing, and I can see it happening right before my eyes lately. This is entirely down to node@6 being out, and Chrome having a large part of ES6 natively available.
As for packaging, I think I've got the my workflow right now. I'll be writing about it in the coming weeks (or months), but it looks like this. Inside of travis (which will get each github push), it'll process all the code to ES5 using Babel, and that is tested, and that is released, but everything locally is written, tested and importantly, run using ES6 - without transpiling.