The article points out that the energy to train BERT is comparable to a flight across America. There are hundreds of thousands of flights every day - why should we be concerned about the equivalent of one extra? Especially given that BERT improves ~10% of English language Google search results, it seems like we're getting a lot in return for relatively small energy use. On top of that, Google buys 100% of their energy usage from Green sources.
I think it's great to talk about other methods of training or architectures that don't require so many parameters. The point about how BERT consumes vastly more text than humans do when they learn to read is interesting. But trying to phrase this like an environmental issue just seems disingenuous and misleading.
I think it's great to talk about other methods of training or architectures that don't require so many parameters. The point about how BERT consumes vastly more text than humans do when they learn to read is interesting. But trying to phrase this like an environmental issue just seems disingenuous and misleading.