Comment by sapphireblue

Meta- reinforcement learning could prove to be such breakthrough, see [1],[2]. Also next generation ASIC accelerators (Google's TPU, Nervana) can give 10x increase in NN performance over a GPU manufactured on the same process, with another 10x possible with some form of binarized weights, e.g. BNN, XNOR-net. There are also interesting techniques to update the model's parameters in a sparse manner.

So, there certainly is a lot of room left for performance improvements!

1. https://arxiv.org/abs/1611.05763 2. https://arxiv.org/abs/1611.02779