A lot of that sounds pretty incredibly wasteful and I don’t think it should be something to strive for more of.
Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong.
Azure charges (like the other major cloud providers) by egress data. I feel bad for whoever is paying the bill for 20gbps going out.
In general, people in ML shouldn’t be pulling raw data in the hundreds of gigs down to mobile workstations. That’s a security and scaling disaster.
Trying not to be too much of a curmudgeon here, but this is the problem with 1 gbps connections. They are so good they prevent you from doing the right architecture until it’s far too late.
A good remote work setup should leave all of the heavy lifting close to the source and anything that does need to be downloaded should be incremental/shallow copies. You should be able to work from a stable coffee shop WiFi connection.
Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong.
Azure charges (like the other major cloud providers) by egress data. I feel bad for whoever is paying the bill for 20gbps going out.
In general, people in ML shouldn’t be pulling raw data in the hundreds of gigs down to mobile workstations. That’s a security and scaling disaster.
Trying not to be too much of a curmudgeon here, but this is the problem with 1 gbps connections. They are so good they prevent you from doing the right architecture until it’s far too late.
A good remote work setup should leave all of the heavy lifting close to the source and anything that does need to be downloaded should be incremental/shallow copies. You should be able to work from a stable coffee shop WiFi connection.