>GPT-4o query consumes around 0.3 watt-hours for a typical text-based question, though this increases substantially to 2.5 to 40 watt-hours for queries with very long inputs.
Given a context window of 128 kilotokens, and how the energy usage range (0.3 to 40 Wh) increases by a factor of 133 from "single query" to "worst case," this suggests linear scaling of energy usage with context length.
Given the common user pattern of "chat with the LLM" the 0.3 Wh figure seems reductive. As the chat grows in length, the length of the "query" becomes the length of the entire chat.
Assuming context compression for long chats, the figure we're looking for per-question energy use should be the energy use for a query half the length of max context tokens.
Given a context window of 128 kilotokens, and how the energy usage range (0.3 to 40 Wh) increases by a factor of 133 from "single query" to "worst case," this suggests linear scaling of energy usage with context length.
Given the common user pattern of "chat with the LLM" the 0.3 Wh figure seems reductive. As the chat grows in length, the length of the "query" becomes the length of the entire chat.
Assuming context compression for long chats, the figure we're looking for per-question energy use should be the energy use for a query half the length of max context tokens.
For GPT-4o: 20 Wh