Comment by dtagames

There is actual, live compute costs every time you submit a prompt, and they're not unsubstantial. If you're not paying for them on your own hardware, you have to rent someone else's hardware to do it on a token-by-token basis or all-you-can-eat-basis. There is no way to save or cache any of this because it all works by predicting what comes next after the entire corpus plus your newly-written prompt. Unless you have that entire model stored, you can't do the calculation.