We had this question on ShopTalk just the other day. We're so (rightfully) worried about web performance, it feels like we shouldn't be loading polyfills all the time, only when we detect they are needed. Philip Walton:
While I understand that this approach is simple, it’s not respectful of your users, and it prioritizes your convenience over their experience.
In a slightly-surprising twist, his tactic is to:
Cut the mustard for browsers that support everything needed
If passes: kick off everything else
If anything fails: load all polyfills then kick things off
The idea is to keep things simple, and not get yourself into a complicated state where you're trying to accommodate for the situation where "the browser can now be in 32 different states".