Comment by gitpusher42

afaik there is some research at this area. Also the new apple foundation model uses related idea. they process the whole prompt and based on prompt load required experts and use only these experts for generation. It doesn't require fitting full model into memory or per token ssd streaming