I wonder if it is possible to have J! not loading all of that.
You could stop using Joomla!. That might work.
Seriously, though, after a quick search I don’t see any J! methods for removing scripts – just for adding them. I don’t use J!, though, so such a function might exist. The alternative is to go through your templates+modules and manually remove them (then make sure everything still works).