Comment by optimalsolver

Is it possible to use derivative-free/black-box optimizers to train these large networks?

From what I understand, gradient descent and its cousins can't suddenly jump to a distant global optimum.