There was an article[1] posted to HN recently about these benchmarks, and it was pretty skeptical.
Regarding SuperGLUE specifically, it asked:
"Indeed, Bowman and his collaborators recently introduced a test called SuperGLUE that's specifically designed to be hard for BERT-based systems. So far, no neural network can beat human performance on it. But even if (or when) it happens, does it mean that machines can really understand language any better than before? Or does just it mean that science has gotten better at teaching machines to the test?"
Regarding SuperGLUE specifically, it asked:
"Indeed, Bowman and his collaborators recently introduced a test called SuperGLUE that's specifically designed to be hard for BERT-based systems. So far, no neural network can beat human performance on it. But even if (or when) it happens, does it mean that machines can really understand language any better than before? Or does just it mean that science has gotten better at teaching machines to the test?"
[1] - https://www.quantamagazine.org/machines-beat-humans-on-a-rea...