When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet.
People read too much into this.
The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sentence-was-on-the-internet).
When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet.
People read too much into this.
The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sentence-was-on-the-internet).