Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.
Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware.
BTW, does anyone knows if LLM Gateway suffers from the same issues? Currently looking at trying it, but haven't got to it yet.