If you give an SLA of lets say 10 seconds to find cheaper inference. I look around different providers that offer flex/batch and try to fulfill in that time.
If I can't then I escalate to a default request.
This means you definitely get a response in 10 seconds (could still be a 5xx or 429 from default), but in those 10 seconds I might find you cheaper inference and drop costs by 50%, then charge 20% of that as FlexInference's cut, effectively giving you a 40% discount.
If I don't find the cheaper inference and have to escalate, then I don't charge at all, so it is free.
If I can't then I escalate to a default request.
This means you definitely get a response in 10 seconds (could still be a 5xx or 429 from default), but in those 10 seconds I might find you cheaper inference and drop costs by 50%, then charge 20% of that as FlexInference's cut, effectively giving you a 40% discount.
If I don't find the cheaper inference and have to escalate, then I don't charge at all, so it is free.