A few technical questions (I had a somewhat related work with friends here https://openreview.net/forum?id=I3HCE7Ro78H although we focused on gradient multiplicity in adversarial training, not massively parallel training)
1. Do you think this is a form of variance reduction or more a form of curriculum (focus first on the bulk, then on remaining errors)?
2. Did you observe any overfitting/additional adversarial risk?
3. Did you try this on just single-node minibatches as well? How did that perform?
1. Do you think this is a form of variance reduction or more a form of curriculum (focus first on the bulk, then on remaining errors)? 2. Did you observe any overfitting/additional adversarial risk? 3. Did you try this on just single-node minibatches as well? How did that perform?