Comment by derivativethrow

Have you read the GPT-3 paper? The authors are pretty forthright about how the GPT techniques likely won't scale much beyond 175B parameters.