> If you instead produce a list of sentences with similar log probabilities you will see that it contains a mix of grammatical and ungrammatical utterances.
Yes, Chomsky mentions this in a footnote. But as far as I know, it hasn't been tried with modern language models.
There's been some interesting work that tries to reproduce grammaticality judgments in terms of language model probability after controlling for length and lexical content. It turns out it works pretty well. For instance https://arxiv.org/pdf/1910.14659.pdf
Yes, Chomsky mentions this in a footnote. But as far as I know, it hasn't been tried with modern language models.
There's been some interesting work that tries to reproduce grammaticality judgments in terms of language model probability after controlling for length and lexical content. It turns out it works pretty well. For instance https://arxiv.org/pdf/1910.14659.pdf