Skimmed a bit and found some snippets, from which I can't take this paper seriously as it dismisses unsupervised learning / language models over large datasets. Yes, sec 4.3.4 briefly discusses recent work in this area, but only briefly and dismisses it by cherry-picking the least positive result of many.
"Only if we have a sufficiently large collection of input-output tuples, in which the
outputs have been appropriately tagged, can we use the data to train a machine so that
it is able, given new inputs sufficiently similar to those in the training data, to predict
corresponding outputs"
This ignores of recent work with large language models that do generalize, zero-shot, to novel tasks.
"supervised learning with core technology end-to-end sequence-to-sequence deep
networks using LSTM (section 4.2.5) with several extensions and variations, including use of GANs"
This reads like something generated from a LM (e.g. GPT-2):
* Where is any mention of attention or Transformer?
* GANs? Have any recent works used GANs successfully for text? There are a few, e.g. CycleGAN, but not widespread afiact.
"Only if we have a sufficiently large collection of input-output tuples, in which the outputs have been appropriately tagged, can we use the data to train a machine so that it is able, given new inputs sufficiently similar to those in the training data, to predict corresponding outputs"
This ignores of recent work with large language models that do generalize, zero-shot, to novel tasks.
"supervised learning with core technology end-to-end sequence-to-sequence deep networks using LSTM (section 4.2.5) with several extensions and variations, including use of GANs"
This reads like something generated from a LM (e.g. GPT-2): * Where is any mention of attention or Transformer? * GANs? Have any recent works used GANs successfully for text? There are a few, e.g. CycleGAN, but not widespread afiact.