- can it be used to relax timing order requirement in pipeline parallelism? Each node update lambda on communication, and optimize weights at other time.
- given that BP is using SGD, can batches and T share the same timeline in optimization, while keeping the descent direction?
- can it be used to relax timing order requirement in pipeline parallelism? Each node update lambda on communication, and optimize weights at other time. - given that BP is using SGD, can batches and T share the same timeline in optimization, while keeping the descent direction?