Comment by CamouflagedKiwi

It's absolutely not for these models. There are plenty of consumer GPUs out there with 8 or 12GB VRAM - they are comparatively very fast at inference but just aren't big enough to run lots of the models you want. Also context management is a massive pain.