Hi everyone. I'm the author. There's one more I thing I wanted to add: a good reason you should try using some sort of hyperparameter search, even you think it's a complete waste of time and compute, is for reproducibility.
This probably applies more to open-source academic contributions, where you're trying to help your fellow practitioners recreate and use your models, as opposed to a corporate setting, where reproducibility would be the equivalent of getting fired.
Recently, I was trying to train a ResNet to beat the top Stanford DAWNBench entry (spoiler alert: I did, but by less than a second). Initially, I blindly tried manually tuning the learning rate, batch size, etc. without even reading the original model's guidelines.
After actually going through a blog post written by the David C Page (the guy with the top DAWNBench entry), I saw that he tried varying the hyperparameters himself and that the ones that were set by default in the code were what he found to be optimal.
That saved me a lot of time and let me focus on other things like what hardware to use.
I think the lesson here is that if more researchers perform and publish the results of some basic hyperparameter optimization, it would really save the world a whole lot of epochs.
This probably applies more to open-source academic contributions, where you're trying to help your fellow practitioners recreate and use your models, as opposed to a corporate setting, where reproducibility would be the equivalent of getting fired.
Recently, I was trying to train a ResNet to beat the top Stanford DAWNBench entry (spoiler alert: I did, but by less than a second). Initially, I blindly tried manually tuning the learning rate, batch size, etc. without even reading the original model's guidelines.
After actually going through a blog post written by the David C Page (the guy with the top DAWNBench entry), I saw that he tried varying the hyperparameters himself and that the ones that were set by default in the code were what he found to be optimal.
That saved me a lot of time and let me focus on other things like what hardware to use.
I think the lesson here is that if more researchers perform and publish the results of some basic hyperparameter optimization, it would really save the world a whole lot of epochs.