Comment by shdjhdfh

He doesn't say anything about individual users. It's left unclear exactly what he means when he talks about low utilization inference using DRAM. I figured he was talking about companies building clusters. I see way more people using old GPUs for local inference than buying 15-20k Mac Studios, or RTX Sparks. I've seen plenty of stories about how companies are building clusters because they want to run open source models, partly for privacy purposes, partly because they're afraid of a rug pull, partly because they want to control when they move to new models, etc, etc.