> As a result, Indeed was able to improve cost and latency by reducing the number of tokens in prompt by 80%.
There's some exact words shenanigans here. Indeed may have reduced the number of tokens in the prompt by 80%, but they didn't reduce the cost by 80%: the prompt cost of inferring from a fine-tuned GPT-3.5-turbo ($3.00 / 1M tokens) is 6x the prompt cost of inferring from the base GPT-3.5-turbo ($0.50 / 1M tokens). If prompt tokens are cut to 20% of normal, then that implies the overall cost of the prompt tokens is 120% relative to their normal prompt: a cost increase! That's not even getting into the 4x cost of the completion tokens for a finetuned model.
Of course, Indeed likely has an enterprise contract reducing costs further.
There's some exact words shenanigans here. Indeed may have reduced the number of tokens in the prompt by 80%, but they didn't reduce the cost by 80%: the prompt cost of inferring from a fine-tuned GPT-3.5-turbo ($3.00 / 1M tokens) is 6x the prompt cost of inferring from the base GPT-3.5-turbo ($0.50 / 1M tokens). If prompt tokens are cut to 20% of normal, then that implies the overall cost of the prompt tokens is 120% relative to their normal prompt: a cost increase! That's not even getting into the 4x cost of the completion tokens for a finetuned model.
Of course, Indeed likely has an enterprise contract reducing costs further.