One of the strangest things in AI, to me, is that you can average the weights of multiple different models to create a single model that’s better than any of the individuals.
This is how distributed training often works, for example. Data parallelism.
I don’t understand why it still works in higher dimensions, but it seems to.
The intuition is that the multiple models are “spinning” around the true solution, so averaging gives the final result more quickly. But it works even early in the training process.
This is how distributed training often works, for example. Data parallelism.
I don’t understand why it still works in higher dimensions, but it seems to.
The intuition is that the multiple models are “spinning” around the true solution, so averaging gives the final result more quickly. But it works even early in the training process.