Comment by numbers_guy

How do they even train such huge models? Especially when you start talking abou multi-modal models. Is the underlying optimization process still relying on gradient descent? I am just baffled how they manage to get it to train across such vastly different sets of data and manage to get it to work.