The underlying mechanism of that is still search. We do have similar mechanism; we can notice it in dreaming, delirium, "word salad" disorder and so on. It will generate formally "new" things and they will look convincingly normal, especially if our defences are down, as in sleep.
I am not sure what the current state of models is, but as far as I know lifting a limit on the model output length tends to noticeably degrade the quality of the result so that it starts to show the same delirious tones.
Fundamentally it is recalling of past experience, a reaction. In many ways it is an opposite of acquiring true new knowledge. We keep that mechanism in check using other human qualities models do not have.
Working with LLM is basically like hiring a madman. It can do useful work, but you cannot trust it. To be economically viable the verification cycle has to be cheap or the cost of an error must be small. Searching a library or doing a home project are fine. But these aren't the lucrative markets investors are dreaming of.
I am not sure what the current state of models is, but as far as I know lifting a limit on the model output length tends to noticeably degrade the quality of the result so that it starts to show the same delirious tones.
Fundamentally it is recalling of past experience, a reaction. In many ways it is an opposite of acquiring true new knowledge. We keep that mechanism in check using other human qualities models do not have.
Working with LLM is basically like hiring a madman. It can do useful work, but you cannot trust it. To be economically viable the verification cycle has to be cheap or the cost of an error must be small. Searching a library or doing a home project are fine. But these aren't the lucrative markets investors are dreaming of.