Comment by bluechair

“What’s the memory bandwidth (GB/s) of the device holding the model weights?”

Isn’t the recommended option going to be dog slow at 256 GB/s.