There are a lot of SSD streaming engines these days. But few to actually try some hard features.
There is one that could really improve the speed. Given almost all major models come with MTP head for speculative decoding. The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.
If the technology demonstrates successful token rate improvement. future models could also come with pretraining heads to preload expert weights, and even make the training be aware of it.
There is one that could really improve the speed. Given almost all major models come with MTP head for speculative decoding. The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.
If the technology demonstrates successful token rate improvement. future models could also come with pretraining heads to preload expert weights, and even make the training be aware of it.