rwitten
Joined 14 years ago
Karma
53
Submissions
19
Member since
Dec 21
Recent submissions
Comment · 3 years ago
Comment · 3 years ago
This is a great series of questions and it isn't our goal to prove you otherwise!
We work with customers interested in training models who run their own ablations, including batch size and learning rates.
Based on that, we demonstrate workloads that we think will be interesting to potential customers! Absolutely agreed that this workload has a larger batch size than the public literature suggests.
Comment · 3 years ago
Great questions! Slices are a set of TPU chips that share a fast, private inter-chip-interconnect. Unlike the current GPU generation in clouds, the TPUs on different machines can communicate through this private network. Multislice means that we're using a hierarchical network, where there is both inter-chip-interconnect and normal data-center netowrking.
More details: https://cloud.google.com/tpu/docs/multislice-introduction
(P.S. - contributor on blog post, Google employee, all thoughts my own)
Comment · 3 years ago
(Contributor on the blog post, all opinions my own)
Agreed with you and we definitely weren't trying to brag! This is fast compared to people's expectations in the space but slow compared to what we should be able to accomplish and will accomplish in the future.
Comment · 3 years ago
Hey! I'm an contributor on this (Rafi Witten), all opinions my own.
You're asking the right question but I think the math is off by a bit. The equivalent number on the H100's is 989 TFLOP/s/chip so the equivalent job is ~10K H100's = (10 * 10^18) / (989 * 10^12). (Both chips also have 8-bit acceleration!)
I believe this is the largest ML job both by exaflops and number of chips every demonstrated. Other companies own more chips or exaflops than we show in this job but getting all the hardware working at once on a single job is a different matter! :-)
Comment · 11 years ago
If you're interested in lower bounds or tighter upper bounds, you can find the latest here:
http://link.springer.com/article/10.1007%2Fs00453-014-9891-7... http://statweb.stanford.edu/~candes/papers/RandomizedNLA.pdf
Totally agreed with all your comments! MaxText mainline is right now a reference implementation for users who have their own scientific opinions on model architecture and convergence. We're additionally hoping to let MaxText run compatibly with some open-source models for customers who want to use known good configurations.