Comment by chargrilled

Bayesian bandits is also a very interesting approach: http://tdunning.blogspot.co.uk/2012/02/bayesian-bandits.html