> So basically, GPT-3 is big, but it's not that good in tasks for which there are formal benchmarks
It is very good in all those tasks. In the paper all the numbers comparing GPT-3 with other models on those benchmark are comparing GPT-3 in a zero/one/few shot setting (so no gradient step), against the previous state of the art finetuned for hours or days on the specific task with millions of gradients steps. If you had time and money to finetue GPT-3 on the specific task there is every reason to believe the gap would be huge.
This is the big thing about GPT-3, the promise of not needing to finetune anymore. This is huge in term of productivity but also because it allows you to use the model in settings for which there are basically no dataset available to finetune on.
It is very good in all those tasks. In the paper all the numbers comparing GPT-3 with other models on those benchmark are comparing GPT-3 in a zero/one/few shot setting (so no gradient step), against the previous state of the art finetuned for hours or days on the specific task with millions of gradients steps. If you had time and money to finetue GPT-3 on the specific task there is every reason to believe the gap would be huge.
This is the big thing about GPT-3, the promise of not needing to finetune anymore. This is huge in term of productivity but also because it allows you to use the model in settings for which there are basically no dataset available to finetune on.