>> If you had time and money to finetue GPT-3 on the specific task there is every reason to believe the gap would be huge.
On the contrary, there is every reason to assume that GPT-3 cannot significantly improve its results on those tasks with extensive fine tuning. Because, if GPT-3 could significantly improve its performance on those tasks with extensive fine-tuning, the OpenAI paper would be reporting those results (edit: OpenAI sure has the time and money to finetune their model).
As we all know by now, there is strong bias in never reporting negative results in machine learning as in other research fields, so we can be reasonably certain that if there is an obvious experiment to perform and that experiment is missing from a paper, it was attemtped and the results were poor.
On the contrary, there is every reason to assume that GPT-3 cannot significantly improve its results on those tasks with extensive fine tuning. Because, if GPT-3 could significantly improve its performance on those tasks with extensive fine-tuning, the OpenAI paper would be reporting those results (edit: OpenAI sure has the time and money to finetune their model).
As we all know by now, there is strong bias in never reporting negative results in machine learning as in other research fields, so we can be reasonably certain that if there is an obvious experiment to perform and that experiment is missing from a paper, it was attemtped and the results were poor.