Currently, LLMs start from scratch for each output token, right?
Lets say you ask an LLM
What makes bananas yellow?
And it replies
Bananas are yellow due to a pigment called bromelain.
I would think that the concept of "pigment" and "bromelain" are already somehow activated in the neural net when it outputs "a". Because now it can't change its mind anymore and follow up with "an optical illusion that makes humans perceive every bent object as yellow". So it seems to have already planned ahead to talk about the pigment called bromelain.
Would it be possible to capitalize on the work that has already been done when the LLM outputs "a"? Could the state of the neural net be somehow preserved for the next answer?
Lets say you ask an LLM
And it replies I would think that the concept of "pigment" and "bromelain" are already somehow activated in the neural net when it outputs "a". Because now it can't change its mind anymore and follow up with "an optical illusion that makes humans perceive every bent object as yellow". So it seems to have already planned ahead to talk about the pigment called bromelain.Would it be possible to capitalize on the work that has already been done when the LLM outputs "a"? Could the state of the neural net be somehow preserved for the next answer?