Comment by orbital-decay

The lack of variety is probably the most underrated issue, and the most likely cause of annoying patterns you all hate. All current post-trained LLMs are mode-collapsed compared to the post-training set distribution, and this doesn't get nearly enough attention from the labs and independent researchers. It's far from being a transformer-only problem, it's a common failure mode in all kinds of models. It's not a fundamental limitation either, unsupervised models have no problem with being as diverse as the training set allows.