dmitrim

Understand, we'll definitely think about the network part. Just in case it may help, if `nvidia-smi nvlink -gt d` is useful for you in this context then there is a related metric NVLink Throughput Rate to compare runs and monitor. At least you might get an idea whether/how internal links are utilized.

For now it only tries to extract NCCL time percentage from the profile, if available, and show it profile summary. Some hints count be in the step trace timeline as well. We are planning to record some NCCL related counters separately as well.

To be more precise, >=2.2 is required for profiler support

Thanks for the feedback, we'll be working on it for sure. At least an explanatory screencast is in the works now, other info material, use cases, etc. are planned.

I'd just use the 2.8, but older 2.x versions should work too. If you encounter are any issues, please let us know via the chat in your account.

Many training and inference workloads run in the cloud or on remote servers and profiling them is not straightforward. Having a SaaS makes things much simpler, and also enables additional features, such as team access and sharing. As far as the data privacy is concerned, profiles do not contain any model or raw data, just resource usage, execution statistics, etc., which is acceptable for most of the users to send to a third party. And for the business model, in my opinion, SaaS allows to better monetize the offering and ensure better and up-to-date end product in this case. But this is open, we may consider a free client-side version as well at some point.