Comment by m0zg

Are you sure this is actually being used for "AutoML" type services? All of the mentioned methods require a parameter search, which is computationally infeasible in a "quick" AutoML use case, and expensive in case you actually need it. That is, you more or less run several training sessions in parallel, and learn from which performs the best in choosing the next parameters. You don't do a full grid search (that's completely unfeasible most of the time), but you at best tweak only a few parameters, and you don't do it every time you train. Hyperparameters aren't just the learning rate and weight decay, it's also the size and number of layers, where and when to quantize and by how much, structure of the network, parameters of pooling, etc etc. I'd say we're still pretty early in the game with all of that, especially when it comes to efficient architectures that demonstrate high accuracy.