Sure, a local model may use a maximum of 65 watts during inference.
However, an H100 runs on 350 watts. Even accounting for the fact that a larger model will need more than one slice, it’s so much faster that this almost balances out.
That’s not the largest difference, though. The H100 can run multiple user queries in parallel, and because LLM inference is limited by memory bandwidth, using large batch sizes this way will dramatically improve efficiency. In short, it takes less power to run it in the datacenter.
Sure, a local model may use a maximum of 65 watts during inference.
However, an H100 runs on 350 watts. Even accounting for the fact that a larger model will need more than one slice, it’s so much faster that this almost balances out.
That’s not the largest difference, though. The H100 can run multiple user queries in parallel, and because LLM inference is limited by memory bandwidth, using large batch sizes this way will dramatically improve efficiency. In short, it takes less power to run it in the datacenter.