Comment by weareconvo

Or to put it mathematically, the optimal amount of exploration time in this explore/exploit problem is far less than you might think.