Comment by moultano

Thank you. Several people have pointed that out, and I'm probably not reading the right papers. Is it common when people introduce a new flavor of adaptive SGD to address how it handles saddles specifically? It is probably just a a matter of what manages to bubble up to me rather than what work is actually getting done, but I felt like the non-convergence of ADAM got talked about a lot, but haven't seen people talking as much about how optimizers behave differently on the landscapes we actually observe.