Comment Comment by brucethemoose2 AFAIK you cannot train 70B on 2x 3090, even with GPTQ/qlora.And the inference is pretty inefficient. Pooling the hardware would achieve much better GPU utilization and (theoretically) faster responses for the host's requests parent brucethemoose2 3 years ago
And the inference is pretty inefficient. Pooling the hardware would achieve much better GPU utilization and (theoretically) faster responses for the host's requests