I am not an expert on why, but my vague answer is you are still running near max clock speeds, and you can significantly downclock your gpu and/or set power limits on your GPU without seeing much loss in performance, e.g. you can cap a 3090 to 60% of it’s normal power draw and it will lose like 5% of tg performance and 10% of pp performance (https://www.reddit.com/r/LocalLLaMA/comments/1hg6qrd/relativ...)
I think theoretically this is still very wasteful with lots of compute that’s getting powered and sitting unused but you are limited either by the firmware or by the GPU architecture from pushing power usage even lower without cratering performance. (no one anticipated the demand for relatively low-compute devices with lots of super fast memory)
I think theoretically this is still very wasteful with lots of compute that’s getting powered and sitting unused but you are limited either by the firmware or by the GPU architecture from pushing power usage even lower without cratering performance. (no one anticipated the demand for relatively low-compute devices with lots of super fast memory)