Personally, I sometimes do this. I even sometimes decide on a per-accesskey base what files are needed. My main reason is to minimize load times. When an admin logs on, they might need JS and CSS for all the admin widgets which a normal user might not even see. So for a normal user, I can leave out all the CSS and JS for those widgets, saving bytes in the process. At the same time, restricting manual access to the files (like just entering the address) may help make attacks more difficult as an attacker has no source code, hence no way of knowing certain interfaces. So they will have to try out a lot more. Using AppSensor, you might be able to detect such behavior and react with a honey pot or aggressive honey pot.
The whole thing has to be implemented server-side, or you will not gain the security advantages. Of course, depending on your implementation, you might also allow for such files to be access either way (which you can do browser-side, take a look at module-loaders, like RequireJS), but personally, because of the security considerations, I would not do something like that.