Comment by rouse

This is really nice. This code looks like it is single-cluster with multiple-GPU. Are there benchmarks on how Horovod work on multi-cluster multi-GPU setting?