A dual Xeon of this era is probably pulling 300W or more when loaded.
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.