I did a quick comparison on MNIST with a small ConvNet, comparing this AdamWSCheduleFree optimizer against a few other optimizers (RAdam, NAdam, AdamW, SGD, Adam, Adafactor, SophiaG). The validation accuracy seems to be okay and the train loss decreases remarkably quickly.
Validation accuracy: https://i.imgur.com/8ZtX7Rd.png
Train loss: https://i.imgur.com/o5XdQ29.png
Code: https://bpa.st/NVJQ (currently only runs on my computer, but not enough time to clean it up)
Note that this is just a toy benchmark with very little hyperparameter tuning. You could probably get similar results with most optimizers and an appropriate schedule. Nevertheless, I appreciate every hyperparameter that I do not have to set manually.
In summary, this seems to be a promising optimizer. I'll add it to my list of optimizers to try for new deep learning projects.
Validation accuracy: https://i.imgur.com/8ZtX7Rd.png
Train loss: https://i.imgur.com/o5XdQ29.png
Code: https://bpa.st/NVJQ (currently only runs on my computer, but not enough time to clean it up)
Note that this is just a toy benchmark with very little hyperparameter tuning. You could probably get similar results with most optimizers and an appropriate schedule. Nevertheless, I appreciate every hyperparameter that I do not have to set manually.
In summary, this seems to be a promising optimizer. I'll add it to my list of optimizers to try for new deep learning projects.