Comment by reilly3000

I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGITS will scratch the itch for local LLM developers, but performant models really want big metal for now, not something I can afford to own or rent at my scale.