Comment by gwern

One major caveat here: the Chinchilla scaling law indicates that most of this compute was wasted. You could've gotten PaLM's performance at a fraction of the cost - Chinchilla approaches PaLM performance but at a fifth the cost.