iamaaditya

In machine learning (especially deep learning or neural networks), the 'training' is done by using Stochastic Gradient Descent. These gradients are computed using Backpropagation. Backpropagation requires you to do a backward pass of your model (typically many layers of neural weights) and thus requires you to keep in memory a lot of intermediate values (called activations). However, if you are doing "inference" that is if the goal is only to get the result but not improve the model, then you don't have to do the backpropagation and thus you don't need to store/save the intermediate values. As the layers and number of parameters in Deep Learning grows, this difference in computation in training vs inference becomes signifiant. In most modern applications of ML, you train once but infer many times, and thus it makes sense to have specialized hardware that is optimized for "inference" at the cost of its inability to do "training".

Love the UI and the simple design. All the best with the product!

Technically it was heat but mostly it was due to (i) Economics, there was less demand for faster clock speed. Otherwise more research could have gone towards solving heat problem. (ii) Each cycle of CPU was more efficient with ability to execute multiple instructions in a single cycle and with more efficient instruction sets.

Surprisingly, power consumption also made huge impact. As tablets and laptops got more popular than desktop battery life became a major concern and thus TDP played major role in research.

Try this fun experiment: Underclock your CPU by half a GHz and see if you notice the difference in your day to day work.

Recently, I have seen O1 being given pretty generously. Couple of people who I know who completed PhDs from mediocre University and less than 10 papers in total have all gotten their O1s. Yes they had to get reference letters and complete all the formality but nothing stellar or extra-ordinary was required.