Comment by TrinaryWorksToo

Have you looked into HuggingFace Accelerate? People have supposedly been able to make the tradeoff with that. Although you still need to download the huge models.